[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240506 15:13:01.334688 19726 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.67.190:33405
I20240506 15:13:01.336210 19726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 15:13:01.337532 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:01.359654 19735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:01.359702 19733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:01.360512 19732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:01.361411 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:02.512784 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:02.513062 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:02.513279 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008382513255 us; error 0 us; skew 500 ppm
I20240506 15:13:02.514181 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:02.520054 19726 webserver.cc:460] Webserver started at http://127.19.67.190:44881/ using document root <none> and password file <none>
I20240506 15:13:02.521018 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:02.521296 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:02.521803 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:02.528445 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/master-0-root/instance:
uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a"
format_stamp: "Formatted at 2024-05-06 15:13:02 on dist-test-slave-8gjm"
I20240506 15:13:02.536810 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240506 15:13:02.542573 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:02.543977 19726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20240506 15:13:02.544385 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/master-0-root
uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a"
format_stamp: "Formatted at 2024-05-06 15:13:02 on dist-test-slave-8gjm"
I20240506 15:13:02.544749 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:02.601387 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:02.602931 19726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 15:13:02.603401 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:02.685199 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.190:33405
I20240506 15:13:02.685307 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.190:33405 every 8 connection(s)
I20240506 15:13:02.693830 19793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:02.720501 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: Bootstrap starting.
I20240506 15:13:02.727144 19793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:02.729084 19793 log.cc:824] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: Log is configured to *not* fsync() on all Append() calls
I20240506 15:13:02.734333 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: No bootstrap required, opened a new log
I20240506 15:13:02.760011 19793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER }
I20240506 15:13:02.760619 19793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:02.760887 19793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7a0ea266964d9a9dcb131bcac9ff7a, State: Initialized, Role: FOLLOWER
I20240506 15:13:02.761768 19793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER }
I20240506 15:13:02.762264 19793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:13:02.762549 19793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:13:02.762882 19793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:02.766978 19793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER }
I20240506 15:13:02.767781 19793 leader_election.cc:302] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7a0ea266964d9a9dcb131bcac9ff7a; no voters:
I20240506 15:13:02.769917 19793 leader_election.cc:288] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:13:02.770316 19796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:02.773964 19796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 1 LEADER]: Becoming Leader. State: Replica: 7f7a0ea266964d9a9dcb131bcac9ff7a, State: Running, Role: LEADER
I20240506 15:13:02.774880 19796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER }
I20240506 15:13:02.775142 19793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:13:02.789505 19798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f7a0ea266964d9a9dcb131bcac9ff7a. Latest consensus state: current_term: 1 leader_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER } }
I20240506 15:13:02.790562 19798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [sys.catalog]: This master's current role is: LEADER
I20240506 15:13:02.790585 19797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a0ea266964d9a9dcb131bcac9ff7a" member_type: VOTER } }
I20240506 15:13:02.791414 19797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [sys.catalog]: This master's current role is: LEADER
I20240506 15:13:02.794497 19806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:13:02.808810 19806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:13:02.810932 19726 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240506 15:13:02.828788 19806 catalog_manager.cc:1347] Generated new cluster ID: 15e9f2dfea7a4088ba5b0bb53a445f4c
I20240506 15:13:02.829128 19806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:13:02.857143 19806 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:13:02.859355 19806 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:13:02.879105 19806 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: Generated new TSK 0
I20240506 15:13:02.880074 19806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:13:02.942678 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:02.952971 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:02.954181 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:02.960381 19726 server_base.cc:1034] running on GCE node
W20240506 15:13:02.962594 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:02.963606 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:02.963841 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:02.964013 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008382963995 us; error 0 us; skew 500 ppm
I20240506 15:13:02.964684 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:02.969921 19726 webserver.cc:460] Webserver started at http://127.19.67.129:35695/ using document root <none> and password file <none>
I20240506 15:13:02.970495 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:02.970707 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:02.970986 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:02.972142 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-0-root/instance:
uuid: "67088c3e2838499e85f0cf4aea0d12c8"
format_stamp: "Formatted at 2024-05-06 15:13:02 on dist-test-slave-8gjm"
I20240506 15:13:02.979285 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240506 15:13:02.984350 19823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:02.985309 19726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240506 15:13:02.985675 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
uuid: "67088c3e2838499e85f0cf4aea0d12c8"
format_stamp: "Formatted at 2024-05-06 15:13:02 on dist-test-slave-8gjm"
I20240506 15:13:02.986063 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:03.001096 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:03.002418 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:03.005211 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:03.009644 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:03.009883 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.010181 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:03.010354 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.131865 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.129:44149
I20240506 15:13:03.131975 19885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.129:44149 every 8 connection(s)
I20240506 15:13:03.148301 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:03.165055 19891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:03.168040 19893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:03.163347 19890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:03.170985 19886 heartbeater.cc:347] Connected to a master server at 127.19.67.190:33405
I20240506 15:13:03.171775 19886 heartbeater.cc:464] Registering TS with master...
I20240506 15:13:03.172976 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:03.173162 19886 heartbeater.cc:511] Master 127.19.67.190:33405 requested a full tablet report, sending...
I20240506 15:13:03.173942 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:03.174225 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:03.174437 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008383174420 us; error 0 us; skew 500 ppm
I20240506 15:13:03.175289 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:03.177682 19758 ts_manager.cc:194] Registered new tserver with Master: 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149)
I20240506 15:13:03.179122 19726 webserver.cc:460] Webserver started at http://127.19.67.130:37765/ using document root <none> and password file <none>
I20240506 15:13:03.179847 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:03.180121 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:03.180038 19758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37554
I20240506 15:13:03.180557 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:03.182102 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-1-root/instance:
uuid: "fa025decf08f4c6689d5dd7b541f980c"
format_stamp: "Formatted at 2024-05-06 15:13:03 on dist-test-slave-8gjm"
I20240506 15:13:03.189306 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240506 15:13:03.194722 19898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.195648 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 15:13:03.195935 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
uuid: "fa025decf08f4c6689d5dd7b541f980c"
format_stamp: "Formatted at 2024-05-06 15:13:03 on dist-test-slave-8gjm"
I20240506 15:13:03.196281 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:03.214931 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:03.216281 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:03.218353 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:03.221668 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:03.221892 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.222131 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:03.222292 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.333858 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.130:37639
I20240506 15:13:03.333992 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.130:37639 every 8 connection(s)
I20240506 15:13:03.345476 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 15:13:03.354745 19961 heartbeater.cc:347] Connected to a master server at 127.19.67.190:33405
I20240506 15:13:03.355464 19961 heartbeater.cc:464] Registering TS with master...
W20240506 15:13:03.355911 19964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:03.356832 19961 heartbeater.cc:511] Master 127.19.67.190:33405 requested a full tablet report, sending...
I20240506 15:13:03.359243 19758 ts_manager.cc:194] Registered new tserver with Master: fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:03.361094 19758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37568
W20240506 15:13:03.363713 19965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:03.366108 19967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:03.368422 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:03.369246 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:03.369429 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:03.369603 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008383369584 us; error 0 us; skew 500 ppm
I20240506 15:13:03.370210 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:03.373049 19726 webserver.cc:460] Webserver started at http://127.19.67.131:34217/ using document root <none> and password file <none>
I20240506 15:13:03.373518 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:03.373688 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:03.373967 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:03.375028 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-2-root/instance:
uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
format_stamp: "Formatted at 2024-05-06 15:13:03 on dist-test-slave-8gjm"
I20240506 15:13:03.380125 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 15:13:03.383540 19972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.384505 19726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 15:13:03.384789 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
format_stamp: "Formatted at 2024-05-06 15:13:03 on dist-test-slave-8gjm"
I20240506 15:13:03.385085 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715008381158361-19726-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:03.407014 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:03.408187 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:03.410243 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:03.413185 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:03.413390 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.413640 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:03.413798 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:03.536823 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.131:34721
I20240506 15:13:03.536904 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.131:34721 every 8 connection(s)
I20240506 15:13:03.555145 20035 heartbeater.cc:347] Connected to a master server at 127.19.67.190:33405
I20240506 15:13:03.555553 20035 heartbeater.cc:464] Registering TS with master...
I20240506 15:13:03.556877 20035 heartbeater.cc:511] Master 127.19.67.190:33405 requested a full tablet report, sending...
I20240506 15:13:03.559494 19758 ts_manager.cc:194] Registered new tserver with Master: 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:03.560097 19726 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016779797s
I20240506 15:13:03.561326 19758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37578
I20240506 15:13:03.563915 19726 test_util.cc:268] Using random seed: -2122499991
I20240506 15:13:03.632992 19758 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37588:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:13:03.638377 19758 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:13:03.755198 20000 tablet_service.cc:1467] Processing CreateTablet for tablet 34b195d05f8246c082221976baa6842f (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240506 15:13:03.764182 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b195d05f8246c082221976baa6842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.759004 19999 tablet_service.cc:1467] Processing CreateTablet for tablet e682ccc2da5640e289ee1efacf2b47f5 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240506 15:13:03.767464 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 49add7ebe9ce4c9a9b607d9298478113 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240506 15:13:03.766347 19995 tablet_service.cc:1467] Processing CreateTablet for tablet 35af3545a5d74d0e99626672ab116d61 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240506 15:13:03.769785 19998 tablet_service.cc:1467] Processing CreateTablet for tablet 63f217906cf442258a45bd798ac4ed03 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240506 15:13:03.775970 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f217906cf442258a45bd798ac4ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.771697 19994 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc75e349edd4cba9b6222666a2b7195 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240506 15:13:03.776839 19997 tablet_service.cc:1467] Processing CreateTablet for tablet a5565a41f820419081e56978575cad4a (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240506 15:13:03.775966 19923 tablet_service.cc:1467] Processing CreateTablet for tablet a5565a41f820419081e56978575cad4a (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240506 15:13:03.781441 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5565a41f820419081e56978575cad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.785334 19999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e682ccc2da5640e289ee1efacf2b47f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.789404 19924 tablet_service.cc:1467] Processing CreateTablet for tablet 63f217906cf442258a45bd798ac4ed03 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240506 15:13:03.787125 19922 tablet_service.cc:1467] Processing CreateTablet for tablet 49add7ebe9ce4c9a9b607d9298478113 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240506 15:13:03.791157 19926 tablet_service.cc:1467] Processing CreateTablet for tablet 34b195d05f8246c082221976baa6842f (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240506 15:13:03.779633 19993 tablet_service.cc:1467] Processing CreateTablet for tablet 135f3f56fc8c4bdf8821262d9d210349 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240506 15:13:03.788623 19992 tablet_service.cc:1467] Processing CreateTablet for tablet 14c3ff62d52c4e2a82761c3458e2c8d3 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240506 15:13:03.794183 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc75e349edd4cba9b6222666a2b7195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.797624 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f217906cf442258a45bd798ac4ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.790460 19925 tablet_service.cc:1467] Processing CreateTablet for tablet e682ccc2da5640e289ee1efacf2b47f5 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240506 15:13:03.801970 19992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c3ff62d52c4e2a82761c3458e2c8d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:13:03.807917 19805 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:13:03.804601 19920 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc75e349edd4cba9b6222666a2b7195 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240506 15:13:03.809020 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49add7ebe9ce4c9a9b607d9298478113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.800204 19921 tablet_service.cc:1467] Processing CreateTablet for tablet 35af3545a5d74d0e99626672ab116d61 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240506 15:13:03.813376 19920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc75e349edd4cba9b6222666a2b7195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.812561 19919 tablet_service.cc:1467] Processing CreateTablet for tablet 135f3f56fc8c4bdf8821262d9d210349 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240506 15:13:03.815974 19921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af3545a5d74d0e99626672ab116d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.818120 19925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e682ccc2da5640e289ee1efacf2b47f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.819506 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5565a41f820419081e56978575cad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.819581 19918 tablet_service.cc:1467] Processing CreateTablet for tablet 14c3ff62d52c4e2a82761c3458e2c8d3 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240506 15:13:03.822314 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f3f56fc8c4bdf8821262d9d210349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.825841 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49add7ebe9ce4c9a9b607d9298478113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.827030 19926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b195d05f8246c082221976baa6842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.828030 19850 tablet_service.cc:1467] Processing CreateTablet for tablet e682ccc2da5640e289ee1efacf2b47f5 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240506 15:13:03.831146 19918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c3ff62d52c4e2a82761c3458e2c8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.837251 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f3f56fc8c4bdf8821262d9d210349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.837119 19849 tablet_service.cc:1467] Processing CreateTablet for tablet 63f217906cf442258a45bd798ac4ed03 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240506 15:13:03.847265 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e682ccc2da5640e289ee1efacf2b47f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.845546 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af3545a5d74d0e99626672ab116d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.866506 19846 tablet_service.cc:1467] Processing CreateTablet for tablet 35af3545a5d74d0e99626672ab116d61 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240506 15:13:03.870090 19851 tablet_service.cc:1467] Processing CreateTablet for tablet 34b195d05f8246c082221976baa6842f (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240506 15:13:03.872506 19845 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc75e349edd4cba9b6222666a2b7195 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240506 15:13:03.876377 19843 tablet_service.cc:1467] Processing CreateTablet for tablet 14c3ff62d52c4e2a82761c3458e2c8d3 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240506 15:13:03.872588 19848 tablet_service.cc:1467] Processing CreateTablet for tablet a5565a41f820419081e56978575cad4a (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240506 15:13:03.878943 19844 tablet_service.cc:1467] Processing CreateTablet for tablet 135f3f56fc8c4bdf8821262d9d210349 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240506 15:13:03.879045 19847 tablet_service.cc:1467] Processing CreateTablet for tablet 49add7ebe9ce4c9a9b607d9298478113 (DEFAULT_TABLE table=test-workload [id=7bcd28474e9c4a3f9a4c83c564551b36]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240506 15:13:03.882063 19849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f217906cf442258a45bd798ac4ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.885345 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f3f56fc8c4bdf8821262d9d210349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.889261 19845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc75e349edd4cba9b6222666a2b7195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.890518 19846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af3545a5d74d0e99626672ab116d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.895334 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b195d05f8246c082221976baa6842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.925375 19843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c3ff62d52c4e2a82761c3458e2c8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.933709 19848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5565a41f820419081e56978575cad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:03.942234 19847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49add7ebe9ce4c9a9b607d9298478113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:04.100183 20054 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.120597 20053 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.124895 20055 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.156062 20054 tablet_bootstrap.cc:654] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.163030 20055 tablet_bootstrap.cc:654] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.168726 20053 tablet_bootstrap.cc:654] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.172508 20055 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.173087 20055 ts_tablet_manager.cc:1367] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.009s
I20240506 15:13:04.176110 20055 raft_consensus.cc:348] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.177181 20055 raft_consensus.cc:374] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.177536 20055 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.178140 20054 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.178745 20054 ts_tablet_manager.cc:1367] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.011s
I20240506 15:13:04.178373 20055 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.182770 20054 raft_consensus.cc:348] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.183346 19886 heartbeater.cc:503] Master 127.19.67.190:33405 was elected leader, sending a full tablet report...
I20240506 15:13:04.183552 20054 raft_consensus.cc:374] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.183851 20054 raft_consensus.cc:729] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.184573 20054 consensus_queue.cc:260] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.195425 20054 ts_tablet_manager.cc:1398] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240506 15:13:04.196372 20035 heartbeater.cc:503] Master 127.19.67.190:33405 was elected leader, sending a full tablet report...
I20240506 15:13:04.196719 20054 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.204334 20054 tablet_bootstrap.cc:654] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.210012 20053 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.216240 20054 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.216708 20053 ts_tablet_manager.cc:1367] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.010s
I20240506 15:13:04.216722 20054 ts_tablet_manager.cc:1367] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20240506 15:13:04.219738 20054 raft_consensus.cc:348] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.220577 20054 raft_consensus.cc:374] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.220889 20054 raft_consensus.cc:729] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.221235 20053 raft_consensus.cc:348] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.221577 20054 consensus_queue.cc:260] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.222165 20053 raft_consensus.cc:374] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.222637 20053 raft_consensus.cc:729] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.223439 20053 consensus_queue.cc:260] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.224702 20054 ts_tablet_manager.cc:1398] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240506 15:13:04.227216 20055 ts_tablet_manager.cc:1398] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.054s user 0.021s sys 0.030s
I20240506 15:13:04.227852 20054 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.228174 20055 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.234982 20055 tablet_bootstrap.cc:654] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.235869 20054 tablet_bootstrap.cc:654] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.246843 20055 tablet_bootstrap.cc:492] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.247323 20055 ts_tablet_manager.cc:1367] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20240506 15:13:04.247751 20054 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.248730 20054 ts_tablet_manager.cc:1367] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.016s
I20240506 15:13:04.250831 20055 raft_consensus.cc:348] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.251585 20055 raft_consensus.cc:374] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.251896 20055 raft_consensus.cc:729] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.251921 20054 raft_consensus.cc:348] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.253259 20054 raft_consensus.cc:374] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.253123 20055 consensus_queue.cc:260] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.253628 20054 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.254412 20054 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.255625 20055 ts_tablet_manager.cc:1398] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20240506 15:13:04.256510 20055 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.257500 20054 ts_tablet_manager.cc:1398] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240506 15:13:04.258430 20054 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.264525 20054 tablet_bootstrap.cc:654] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.263548 20055 tablet_bootstrap.cc:654] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.272991 19961 heartbeater.cc:503] Master 127.19.67.190:33405 was elected leader, sending a full tablet report...
I20240506 15:13:04.277691 20053 ts_tablet_manager.cc:1398] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.060s user 0.008s sys 0.001s
I20240506 15:13:04.278648 20053 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.283641 20053 tablet_bootstrap.cc:654] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.288328 20054 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.288823 20054 ts_tablet_manager.cc:1367] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.005s
I20240506 15:13:04.289675 20055 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.290120 20055 ts_tablet_manager.cc:1367] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.003s
I20240506 15:13:04.291751 20054 raft_consensus.cc:348] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.292760 20054 raft_consensus.cc:374] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.293049 20054 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.293637 20055 raft_consensus.cc:348] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.293722 20054 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.294379 20055 raft_consensus.cc:374] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.294673 20055 raft_consensus.cc:729] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.295917 20055 consensus_queue.cc:260] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
W20240506 15:13:04.296756 20036 tablet.cc:2369] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:13:04.298894 20054 ts_tablet_manager.cc:1398] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240506 15:13:04.299818 20054 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.304452 20055 ts_tablet_manager.cc:1398] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.014s user 0.006s sys 0.004s
I20240506 15:13:04.305406 20055 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.312050 20055 tablet_bootstrap.cc:654] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.317272 20053 tablet_bootstrap.cc:492] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.317790 20053 ts_tablet_manager.cc:1367] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.006s
I20240506 15:13:04.319132 20055 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.312652 20054 tablet_bootstrap.cc:654] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.320647 20055 ts_tablet_manager.cc:1367] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20240506 15:13:04.320652 20053 raft_consensus.cc:348] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.321609 20053 raft_consensus.cc:374] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.321918 20053 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.322760 20053 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.326373 20053 ts_tablet_manager.cc:1398] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240506 15:13:04.327455 20053 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.327217 20055 raft_consensus.cc:348] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.328130 20055 raft_consensus.cc:374] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.328537 20055 raft_consensus.cc:729] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.329269 20055 consensus_queue.cc:260] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.334138 20055 ts_tablet_manager.cc:1398] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.013s user 0.000s sys 0.005s
I20240506 15:13:04.334393 20054 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.334820 20054 ts_tablet_manager.cc:1367] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.006s
I20240506 15:13:04.335067 20055 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.340267 20053 tablet_bootstrap.cc:654] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.342521 20055 tablet_bootstrap.cc:654] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.344344 20054 raft_consensus.cc:348] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.345124 20054 raft_consensus.cc:374] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.348990 20054 raft_consensus.cc:729] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.349970 20054 consensus_queue.cc:260] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.352793 20054 ts_tablet_manager.cc:1398] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20240506 15:13:04.352896 20055 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.353335 20055 ts_tablet_manager.cc:1367] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20240506 15:13:04.356539 20055 raft_consensus.cc:348] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.357134 20054 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.357280 20055 raft_consensus.cc:374] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.357581 20055 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.358253 20055 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.363055 20055 ts_tablet_manager.cc:1398] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240506 15:13:04.364205 20055 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.368212 20053 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.369335 20053 ts_tablet_manager.cc:1367] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.015s
I20240506 15:13:04.372875 20054 tablet_bootstrap.cc:654] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.378214 20055 tablet_bootstrap.cc:654] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.379905 20053 raft_consensus.cc:348] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.380518 20053 raft_consensus.cc:374] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.380753 20053 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.381577 20053 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.385113 20053 ts_tablet_manager.cc:1398] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.015s user 0.000s sys 0.008s
I20240506 15:13:04.386888 20053 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.395483 20054 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.395999 20054 ts_tablet_manager.cc:1367] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.004s
I20240506 15:13:04.396657 20053 tablet_bootstrap.cc:654] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.397540 20055 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.398275 20055 ts_tablet_manager.cc:1367] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.008s
I20240506 15:13:04.400138 20054 raft_consensus.cc:348] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.401126 20054 raft_consensus.cc:374] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.402161 20055 raft_consensus.cc:348] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.403061 20055 raft_consensus.cc:374] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.403406 20055 raft_consensus.cc:729] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.401451 20054 raft_consensus.cc:729] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.405308 20059 raft_consensus.cc:482] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.404383 20055 consensus_queue.cc:260] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.406677 20059 raft_consensus.cc:504] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.410282 20059 leader_election.cc:288] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.411520 20054 consensus_queue.cc:260] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.414263 20054 ts_tablet_manager.cc:1398] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20240506 15:13:04.415158 20053 tablet_bootstrap.cc:492] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.415212 20054 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.415792 20053 ts_tablet_manager.cc:1367] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.016s
I20240506 15:13:04.419258 20059 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.419732 20059 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.420352 20053 raft_consensus.cc:348] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.421057 20053 raft_consensus.cc:374] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.421895 20059 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.421866 20053 raft_consensus.cc:729] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.422734 20053 consensus_queue.cc:260] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.425180 20053 ts_tablet_manager.cc:1398] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20240506 15:13:04.426656 20053 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.430347 20059 raft_consensus.cc:482] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.430941 20059 raft_consensus.cc:504] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.433108 20053 tablet_bootstrap.cc:654] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.433073 20059 leader_election.cc:288] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.444607 20054 tablet_bootstrap.cc:654] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.457581 20060 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.456727 20055 ts_tablet_manager.cc:1398] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.058s user 0.008s sys 0.006s
I20240506 15:13:04.458613 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.457983 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.458250 20060 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.462553 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.462116 19935 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.464855 20055 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
W20240506 15:13:04.469558 19826 leader_election.cc:341] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.473850 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.474377 20060 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.476467 20060 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.467336 20008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.475945 20009 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.478446 20008 raft_consensus.cc:2454] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.474059 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.475462 19826 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.482204 20053 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.478873 20060 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.491943 20075 raft_consensus.cc:482] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.491384 20053 ts_tablet_manager.cc:1367] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.015s
I20240506 15:13:04.492847 20075 raft_consensus.cc:504] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.505117 20075 leader_election.cc:288] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.507838 20060 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.508422 20054 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.508374 20060 raft_consensus.cc:482] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.508976 20054 ts_tablet_manager.cc:1367] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.094s user 0.007s sys 0.014s
I20240506 15:13:04.508904 20060 raft_consensus.cc:504] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.510886 20060 leader_election.cc:288] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.493196 20059 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.511947 20054 raft_consensus.cc:348] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.512436 20059 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.512826 20059 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.498718 20055 tablet_bootstrap.cc:654] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.515215 20054 raft_consensus.cc:374] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.528455 20054 raft_consensus.cc:729] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.523473 20053 raft_consensus.cc:348] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.529711 20053 raft_consensus.cc:374] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.529610 20054 consensus_queue.cc:260] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.530354 20053 raft_consensus.cc:729] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.531958 20053 consensus_queue.cc:260] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.541884 20053 ts_tablet_manager.cc:1398] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20240506 15:13:04.534412 20054 ts_tablet_manager.cc:1398] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
W20240506 15:13:04.544924 19826 leader_election.cc:341] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.549923 20053 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.549433 20060 raft_consensus.cc:482] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.559608 20062 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.588578 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.548966 20054 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.591190 19934 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.592669 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.593284 19826 leader_election.cc:302] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.587949 20060 raft_consensus.cc:504] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.588829 20062 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.594760 20075 raft_consensus.cc:2781] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.595144 20075 raft_consensus.cc:482] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.595788 20075 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.594126 19824 leader_election.cc:302] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters: fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.598712 20010 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.583586 20059 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.606377 20075 raft_consensus.cc:504] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.616914 20060 leader_election.cc:288] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.623009 20075 leader_election.cc:288] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.623108 20053 tablet_bootstrap.cc:654] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.621232 20062 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.624501 20059 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.624727 20075 raft_consensus.cc:2781] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.634308 20075 raft_consensus.cc:482] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.635744 20075 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.648506 20054 tablet_bootstrap.cc:654] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.658095 20059 raft_consensus.cc:482] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.658962 20059 raft_consensus.cc:504] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.663075 20059 leader_election.cc:288] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
W20240506 15:13:04.656634 19824 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:34721, user_credentials={real_user=slave}} blocked reactor thread for 63989.4us
I20240506 15:13:04.672428 20055 tablet_bootstrap.cc:492] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.672957 20055 ts_tablet_manager.cc:1367] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.210s user 0.022s sys 0.024s
W20240506 15:13:04.675112 19824 leader_election.cc:341] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.679272 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.680861 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.681619 19935 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.680186 19934 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.676017 19824 leader_election.cc:302] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8; no voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.684204 20060 raft_consensus.cc:482] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.686899 20059 raft_consensus.cc:2726] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:04.687052 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.686924 20060 raft_consensus.cc:504] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.690970 20060 leader_election.cc:288] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.680801 20075 raft_consensus.cc:504] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.689088 19936 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.698717 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.700198 20075 leader_election.cc:288] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.701300 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.701138 20055 raft_consensus.cc:348] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.705945 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:04.708724 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.709038 20010 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.710196 19861 raft_consensus.cc:2379] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a2eb00bc6d8498e99de9079eec2f7de in current term 1: Already voted for candidate 67088c3e2838499e85f0cf4aea0d12c8 in this term.
I20240506 15:13:04.710907 20008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:04.710292 20053 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.711694 20008 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.711632 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.706004 20055 raft_consensus.cc:374] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.712857 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.714249 20009 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:04.715009 20055 raft_consensus.cc:729] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.714231 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:04.715651 20007 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.715911 19824 leader_election.cc:302] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:04.713526 20053 ts_tablet_manager.cc:1367] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.164s user 0.020s sys 0.007s
I20240506 15:13:04.719592 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.715766 20055 consensus_queue.cc:260] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.717378 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.720328 20075 raft_consensus.cc:2781] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.724179 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.728791 19934 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:04.730473 20054 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.731056 20054 ts_tablet_manager.cc:1367] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.182s user 0.018s sys 0.021s
I20240506 15:13:04.723788 20075 raft_consensus.cc:482] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.724452 19936 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:04.733192 19975 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de; no voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c
W20240506 15:13:04.736001 19975 leader_election.cc:341] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.726415 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.732781 20075 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.733816 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.737522 19935 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:04.724232 19859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.740661 20055 ts_tablet_manager.cc:1398] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.067s user 0.010s sys 0.000s
I20240506 15:13:04.740427 20054 raft_consensus.cc:348] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.741209 19859 raft_consensus.cc:2379] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a2eb00bc6d8498e99de9079eec2f7de in current term 1: Already voted for candidate 67088c3e2838499e85f0cf4aea0d12c8 in this term.
I20240506 15:13:04.741129 20054 raft_consensus.cc:374] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.741554 20054 raft_consensus.cc:729] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.741679 20055 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
I20240506 15:13:04.742226 20054 consensus_queue.cc:260] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
W20240506 15:13:04.742918 19974 leader_election.cc:341] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.743197 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.743418 19974 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de; no voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.743849 19861 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 0.
I20240506 15:13:04.744609 19858 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:04.742836 19933 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:04.745893 20082 raft_consensus.cc:2726] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:04.747743 19975 leader_election.cc:302] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.751560 20082 raft_consensus.cc:2781] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.752795 20054 ts_tablet_manager.cc:1398] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20240506 15:13:04.752588 20082 raft_consensus.cc:482] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.754063 20054 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: Bootstrap starting.
I20240506 15:13:04.736541 20053 raft_consensus.cc:348] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.751869 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.754737 20082 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.755502 19826 leader_election.cc:302] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.755676 20053 raft_consensus.cc:374] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.758353 19826 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.759586 20059 raft_consensus.cc:2781] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.763188 19900 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.759276 20053 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.760958 20059 raft_consensus.cc:686] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:04.765108 20062 raft_consensus.cc:2781] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.768803 20062 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.770521 20062 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.768946 20053 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.770265 20059 consensus_queue.cc:237] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.773998 19974 leader_election.cc:302] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de; no voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.770275 20086 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election won for term 1
W20240506 15:13:04.783109 19826 leader_election.cc:341] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.777386 19975 leader_election.cc:302] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.794082 20086 raft_consensus.cc:686] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:04.809944 20075 raft_consensus.cc:504] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.811947 20053 ts_tablet_manager.cc:1398] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.094s user 0.018s sys 0.015s
W20240506 15:13:04.812106 19805 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:13:04.812536 20010 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:04.813190 19860 raft_consensus.cc:2379] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a2eb00bc6d8498e99de9079eec2f7de in current term 1: Already voted for candidate 67088c3e2838499e85f0cf4aea0d12c8 in this term.
I20240506 15:13:04.813516 20075 leader_election.cc:288] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.814277 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.814862 19936 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.829423 20053 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.815240 20086 consensus_queue.cc:237] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.834867 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:04.836232 20008 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:04.836699 20060 raft_consensus.cc:2726] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:04.833894 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:04.838302 20007 raft_consensus.cc:2454] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
W20240506 15:13:04.843115 19899 leader_election.cc:341] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.845934 19824 leader_election.cc:302] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters: fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.847111 20075 raft_consensus.cc:2781] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.847587 20075 raft_consensus.cc:686] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:04.845541 20055 tablet_bootstrap.cc:654] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.843310 20082 raft_consensus.cc:504] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.850186 19936 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:04.848723 20075 consensus_queue.cc:237] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.856567 20054 tablet_bootstrap.cc:654] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.856942 19826 leader_election.cc:302] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.858887 20009 raft_consensus.cc:2379] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67088c3e2838499e85f0cf4aea0d12c8 in current term 1: Already voted for candidate 4a2eb00bc6d8498e99de9079eec2f7de in this term.
I20240506 15:13:04.860949 20096 raft_consensus.cc:2726] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:04.861162 20093 raft_consensus.cc:2781] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.859305 20062 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.867702 20093 raft_consensus.cc:686] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:04.860978 19755 catalog_manager.cc:5484] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 1 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:04.869376 20095 raft_consensus.cc:2781] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.872298 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:04.872902 20062 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:04.872504 20095 raft_consensus.cc:482] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.874164 20095 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.873907 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:04.874282 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
W20240506 15:13:04.877420 19899 leader_election.cc:341] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.875087 19858 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.870149 20093 consensus_queue.cc:237] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.879254 20082 leader_election.cc:288] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.881141 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.887255 19936 raft_consensus.cc:2379] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a2eb00bc6d8498e99de9079eec2f7de in current term 1: Already voted for candidate 67088c3e2838499e85f0cf4aea0d12c8 in this term.
I20240506 15:13:04.890834 19975 leader_election.cc:302] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de; no voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.892501 20082 raft_consensus.cc:2726] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:13:04.893397 20053 tablet_bootstrap.cc:654] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.897806 20054 tablet_bootstrap.cc:492] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: No bootstrap required, opened a new log
I20240506 15:13:04.898172 20054 ts_tablet_manager.cc:1367] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.007s
I20240506 15:13:04.902281 19858 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 1.
I20240506 15:13:04.903551 20054 raft_consensus.cc:348] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.905241 20054 raft_consensus.cc:374] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.905649 20054 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.906855 19900 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters: 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:04.908124 20062 raft_consensus.cc:2781] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.907678 20054 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.908638 20062 raft_consensus.cc:686] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:04.909685 20062 consensus_queue.cc:237] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.909685 20055 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.913179 20055 ts_tablet_manager.cc:1367] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.172s user 0.022s sys 0.006s
I20240506 15:13:04.915527 20055 raft_consensus.cc:348] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.916054 20055 raft_consensus.cc:374] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.916342 20055 raft_consensus.cc:729] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.916911 20055 consensus_queue.cc:260] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.912606 20054 ts_tablet_manager.cc:1398] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20240506 15:13:04.921826 20082 raft_consensus.cc:482] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.922523 20082 raft_consensus.cc:504] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.923127 20055 ts_tablet_manager.cc:1398] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240506 15:13:04.926194 20082 leader_election.cc:288] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.928113 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.928653 20055 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: Bootstrap starting.
W20240506 15:13:04.929934 19975 leader_election.cc:341] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.931685 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.934880 19858 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:04.934196 19757 catalog_manager.cc:5484] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 0 to 1, leader changed from <none> to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 1 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:04.922179 20095 raft_consensus.cc:504] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.936506 19974 leader_election.cc:302] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters: fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.937367 20082 raft_consensus.cc:2781] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.937819 20082 raft_consensus.cc:482] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.938344 20082 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.939352 20053 tablet_bootstrap.cc:492] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:04.940539 20053 ts_tablet_manager.cc:1367] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.111s user 0.015s sys 0.016s
I20240506 15:13:04.946439 20055 tablet_bootstrap.cc:654] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.950330 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.950569 20053 raft_consensus.cc:348] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.951064 19936 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.951467 20053 raft_consensus.cc:374] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.951898 20053 raft_consensus.cc:729] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.949735 20095 leader_election.cc:288] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.951251 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:04.953307 20053 consensus_queue.cc:260] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.954838 20082 raft_consensus.cc:504] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.956367 20053 ts_tablet_manager.cc:1398] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240506 15:13:04.956871 20082 leader_election.cc:288] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.957515 20053 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:04.957989 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:04.958369 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:04.958603 19858 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:13:04.960733 19975 leader_election.cc:341] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:04.958606 19860 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.961748 19936 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 1.
I20240506 15:13:04.965899 19975 leader_election.cc:302] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.967026 20095 raft_consensus.cc:482] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:04.967496 20095 raft_consensus.cc:504] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.968948 19860 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 1.
I20240506 15:13:04.970036 20082 raft_consensus.cc:2781] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.970919 20053 tablet_bootstrap.cc:654] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:04.970830 20082 raft_consensus.cc:686] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:04.970829 19974 leader_election.cc:302] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters: fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:04.972159 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:04.972652 20096 raft_consensus.cc:2781] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:04.972066 20082 consensus_queue.cc:237] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.973349 20096 raft_consensus.cc:686] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:04.975824 20095 leader_election.cc:288] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:04.975695 20096 consensus_queue.cc:237] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.980052 20055 tablet_bootstrap.cc:492] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: No bootstrap required, opened a new log
I20240506 15:13:04.980544 20055 ts_tablet_manager.cc:1367] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.004s
I20240506 15:13:04.973194 19936 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:04.982400 19975 leader_election.cc:302] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:04.983477 20095 raft_consensus.cc:2781] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:04.983512 20055 raft_consensus.cc:348] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.983984 20095 raft_consensus.cc:482] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:04.984375 20055 raft_consensus.cc:374] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:04.984442 20095 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:04.984723 20055 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Initialized, Role: FOLLOWER
I20240506 15:13:04.986308 20055 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:04.989418 20055 ts_tablet_manager.cc:1398] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240506 15:13:04.991168 19858 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 1.
I20240506 15:13:04.995621 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:04.996639 19860 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 0.
I20240506 15:13:05.000844 20095 raft_consensus.cc:504] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.009788 20053 tablet_bootstrap.cc:492] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:05.010145 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.010762 19860 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.010308 20053 ts_tablet_manager.cc:1367] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.013s
I20240506 15:13:05.015291 20053 raft_consensus.cc:348] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.016022 20053 raft_consensus.cc:374] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:05.016317 20053 raft_consensus.cc:729] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:05.016954 20053 consensus_queue.cc:260] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.019109 20053 ts_tablet_manager.cc:1398] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240506 15:13:05.019109 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:05.019858 19936 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.020364 20053 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: Bootstrap starting.
I20240506 15:13:05.027123 20095 leader_election.cc:288] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:05.030157 19860 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 1.
I20240506 15:13:05.031174 19756 catalog_manager.cc:5484] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 1 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:05.032397 19974 leader_election.cc:302] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:05.028888 19936 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 1.
I20240506 15:13:05.034852 20053 tablet_bootstrap.cc:654] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:05.032852 19755 catalog_manager.cc:5484] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 1 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.041174 19755 catalog_manager.cc:5484] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 1 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.042923 20095 raft_consensus.cc:2781] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:05.043666 20095 raft_consensus.cc:686] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:05.046753 20053 tablet_bootstrap.cc:492] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: No bootstrap required, opened a new log
I20240506 15:13:05.047150 20053 ts_tablet_manager.cc:1367] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.004s
I20240506 15:13:05.046337 19755 catalog_manager.cc:5484] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 1 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.044565 20095 consensus_queue.cc:237] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.049328 20053 raft_consensus.cc:348] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.049883 20053 raft_consensus.cc:374] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:05.050109 20053 raft_consensus.cc:729] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Initialized, Role: FOLLOWER
I20240506 15:13:05.050627 20053 consensus_queue.cc:260] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.052083 20053 ts_tablet_manager.cc:1398] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 15:13:05.066648 19756 catalog_manager.cc:5484] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 1 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:05.068703 19756 catalog_manager.cc:5484] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 1 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:05.109477 20093 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:05.110090 20093 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.113190 20093 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.113395 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:05.114177 19936 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:05.114315 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:05.114928 20009 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 0.
I20240506 15:13:05.115242 19826 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.116031 20093 raft_consensus.cc:2781] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:05.116421 20093 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:05.116755 20093 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.122663 20062 raft_consensus.cc:482] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:05.123106 20062 raft_consensus.cc:504] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.123296 20093 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.125239 20093 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.125375 20062 leader_election.cc:288] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.126425 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:05.127152 19936 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.127329 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" is_pre_election: true
I20240506 15:13:05.127648 20008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.126479 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:05.128235 20008 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.129374 19860 raft_consensus.cc:2379] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa025decf08f4c6689d5dd7b541f980c in current term 1: Already voted for candidate 4a2eb00bc6d8498e99de9079eec2f7de in this term.
I20240506 15:13:05.130357 19900 leader_election.cc:302] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa025decf08f4c6689d5dd7b541f980c; no voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8
I20240506 15:13:05.131385 20062 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.136708 19936 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:05.137428 20008 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 1.
I20240506 15:13:05.138224 19826 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.138733 20062 raft_consensus.cc:2726] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:05.139099 20093 raft_consensus.cc:2781] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:05.139490 20095 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:05.139627 20093 raft_consensus.cc:686] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:05.139989 20095 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.141073 20093 consensus_queue.cc:237] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.142235 20095 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149)
I20240506 15:13:05.143229 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa025decf08f4c6689d5dd7b541f980c" is_pre_election: true
I20240506 15:13:05.143476 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8" is_pre_election: true
I20240506 15:13:05.145103 19860 raft_consensus.cc:2379] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a2eb00bc6d8498e99de9079eec2f7de in current term 1: Already voted for candidate fa025decf08f4c6689d5dd7b541f980c in this term.
I20240506 15:13:05.146445 19974 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de; no voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:05.147552 20095 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.154897 20095 raft_consensus.cc:2726] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:05.155042 19756 catalog_manager.cc:5484] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 1 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.234012 19936 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:33998
I20240506 15:13:05.234843 19936 raft_consensus.cc:595] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 LEADER]: Received request to transfer leadership to TS 67088c3e2838499e85f0cf4aea0d12c8
I20240506 15:13:05.270403 19860 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:51550
I20240506 15:13:05.270922 19860 raft_consensus.cc:595] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:05.284492 20008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:32992
I20240506 15:13:05.285027 20008 raft_consensus.cc:595] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:05.290547 20008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:32992
I20240506 15:13:05.291199 20008 raft_consensus.cc:595] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:05.296618 20008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:32992
I20240506 15:13:05.297237 20008 raft_consensus.cc:595] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:05.303059 19860 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:51550
I20240506 15:13:05.303576 19860 raft_consensus.cc:595] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Received request to transfer leadership to TS 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:05.313225 20093 consensus_queue.cc:1035] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.324296 20062 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.331581 20059 consensus_queue.cc:1035] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.338326 20094 raft_consensus.cc:979] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:05.338794 20094 raft_consensus.cc:1067] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 LEADER]: Signalling peer 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:05.342103 19858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:51548
I20240506 15:13:05.342733 19858 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.343104 19858 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.347864 19933 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:05.351148 19858 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.363232 20095 consensus_queue.cc:1035] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:05.363770 19860 raft_consensus.cc:1226] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Rejecting Update request from peer fa025decf08f4c6689d5dd7b541f980c for earlier term 1. Current term is 2. Ops: []
I20240506 15:13:05.366125 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.358429 20094 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.366746 20007 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.357714 20075 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.371699 19858 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 2 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.376747 20007 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 2.
I20240506 15:13:05.377665 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:05.378348 19935 raft_consensus.cc:3032] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.378738 19935 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:05.379621 19935 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.382565 19935 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.387907 20059 consensus_queue.cc:1035] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.390370 20010 raft_consensus.cc:1226] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Rejecting Update request from peer fa025decf08f4c6689d5dd7b541f980c for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240506 15:13:05.392781 19824 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:05.392287 20059 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:05.393761 20075 raft_consensus.cc:2781] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.402082 20075 raft_consensus.cc:686] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:05.403218 20075 consensus_queue.cc:237] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.404537 20096 consensus_queue.cc:1035] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.410600 19935 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 2.
I20240506 15:13:05.412567 19757 catalog_manager.cc:5484] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 1 to 2, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 2 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.416311 20075 consensus_queue.cc:1035] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.433264 20059 consensus_queue.cc:1035] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.449295 20096 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.452708 20096 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:05.464910 20128 consensus_queue.cc:1035] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.482858 20060 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.497772 20060 raft_consensus.cc:979] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.498243 20060 raft_consensus.cc:1067] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.500522 19931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33996
I20240506 15:13:05.501961 19931 raft_consensus.cc:482] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.502408 19931 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.507982 20095 raft_consensus.cc:979] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.508522 20095 raft_consensus.cc:1067] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.509802 20075 raft_consensus.cc:979] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:05.510267 20075 raft_consensus.cc:1067] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Signalling peer 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:05.517251 19930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33996
I20240506 15:13:05.517988 19930 raft_consensus.cc:482] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.518445 19930 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.517266 20010 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:32970
I20240506 15:13:05.534633 20060 raft_consensus.cc:979] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.535183 20060 raft_consensus.cc:1067] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.539466 20010 raft_consensus.cc:482] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.539950 20010 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.541603 19935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33996
I20240506 15:13:05.544580 20093 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:05.553042 20095 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.563081 20010 raft_consensus.cc:504] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.565013 19930 raft_consensus.cc:504] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.566162 19934 raft_consensus.cc:1226] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Rejecting Update request from peer 4a2eb00bc6d8498e99de9079eec2f7de for earlier term 1. Current term is 2. Ops: []
I20240506 15:13:05.578140 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.578804 19861 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.580754 20093 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:05.588985 19930 leader_election.cc:288] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.589128 19861 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.589789 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.590564 20009 raft_consensus.cc:3032] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.590826 19900 leader_election.cc:302] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.591001 20009 raft_consensus.cc:729] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:05.591742 20094 raft_consensus.cc:2781] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.592242 20094 raft_consensus.cc:686] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:05.592442 20009 consensus_queue.cc:260] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.593528 20094 consensus_queue.cc:237] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.594460 20009 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.603264 20128 raft_consensus.cc:979] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.603761 20128 raft_consensus.cc:1067] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:05.605759 19930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33982
I20240506 15:13:05.606395 19930 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.606901 19930 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.554634 19931 raft_consensus.cc:504] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.611050 19935 raft_consensus.cc:482] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:05.611513 19935 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.612713 20010 leader_election.cc:288] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:05.614166 19931 leader_election.cc:288] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.616053 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.616693 19860 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.618402 19935 raft_consensus.cc:504] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.619237 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.619863 19861 raft_consensus.cc:3032] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.620195 19861 raft_consensus.cc:729] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:05.624047 20009 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.622026 19930 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.621465 19861 consensus_queue.cc:260] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.630156 19861 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.631891 19930 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.632676 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f3f56fc8c4bdf8821262d9d210349" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.633714 20009 raft_consensus.cc:3032] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.633659 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.634284 20009 raft_consensus.cc:729] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:05.634495 19858 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.635224 20009 consensus_queue.cc:260] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.626036 19935 leader_election.cc:288] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:05.638062 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.638728 20010 raft_consensus.cc:3032] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.639101 20010 raft_consensus.cc:729] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:05.639897 20010 consensus_queue.cc:260] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.640580 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c3ff62d52c4e2a82761c3458e2c8d3" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:05.641218 19934 raft_consensus.cc:3037] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.641284 20010 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.643015 19860 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.645663 20009 raft_consensus.cc:3037] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.645638 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:05.646853 20007 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.647668 20010 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.649061 19861 raft_consensus.cc:2454] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 2.
I20240506 15:13:05.648974 19899 leader_election.cc:302] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.650832 19900 leader_election.cc:302] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.651575 20062 raft_consensus.cc:2781] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.657526 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:05.658387 19861 raft_consensus.cc:3032] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:05.657938 19974 leader_election.cc:302] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:05.658836 19861 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:05.659507 20060 raft_consensus.cc:2781] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.660044 20060 raft_consensus.cc:686] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:05.659910 19861 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.661075 20060 consensus_queue.cc:237] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.661789 19861 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:05.653791 20094 raft_consensus.cc:2781] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.672086 20007 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.674556 20009 raft_consensus.cc:2454] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.679466 19899 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:05.681116 20062 raft_consensus.cc:686] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:05.682765 20062 consensus_queue.cc:237] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.679731 19861 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.690624 20143 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:05.691387 20143 raft_consensus.cc:686] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:05.692546 20143 consensus_queue.cc:237] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.688884 19934 raft_consensus.cc:2454] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 2.
I20240506 15:13:05.689338 19858 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:05.702785 19755 catalog_manager.cc:5484] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 1 to 2, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 2 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:05.706233 20094 raft_consensus.cc:686] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:05.707228 20094 consensus_queue.cc:237] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:05.721619 19755 catalog_manager.cc:5484] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.723883 19755 catalog_manager.cc:5484] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.726125 19755 catalog_manager.cc:5484] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.728468 19755 catalog_manager.cc:5484] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:05.828507 19934 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:05.829694 20093 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:05.839180 20009 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:05.840938 20128 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:06.005008 20009 raft_consensus.cc:1261] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.007009 20094 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:13:06.016821 19858 raft_consensus.cc:1261] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.018448 20094 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.065128 19858 raft_consensus.cc:1261] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.066223 19934 raft_consensus.cc:1261] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.066550 20060 consensus_queue.cc:1035] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.067605 20096 consensus_queue.cc:1035] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.190654 19858 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.192183 20094 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.192637 20009 raft_consensus.cc:1261] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.194248 20094 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:13:06.204387 19858 raft_consensus.cc:1261] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.205642 20009 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.206497 20143 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.208765 20145 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.315578 20007 raft_consensus.cc:1261] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.316936 20145 consensus_queue.cc:1035] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:06.326608 19858 raft_consensus.cc:1261] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:06.328130 20144 consensus_queue.cc:1035] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:08.327003 19726 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240506 15:13:08.327744 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:08.349490 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:08.350158 19934 raft_consensus.cc:595] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Received request to transfer leadership to TS 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:08.360653 19858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:51558
I20240506 15:13:08.361184 19858 raft_consensus.cc:595] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Received request to transfer leadership to TS 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:08.362274 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240506 15:13:08.362701 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:08.383616 20144 raft_consensus.cc:979] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:08.384054 20148 raft_consensus.cc:1067] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Signalling peer 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:08.385497 20007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:32986
I20240506 15:13:08.386006 20007 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:08.386301 20007 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:08.390619 20007 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.392149 20007 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 3 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:08.393051 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:08.393142 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:08.393685 19858 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:08.393775 19934 raft_consensus.cc:3032] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:08.394102 19934 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:08.394691 19934 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.395566 19934 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:08.398141 19858 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 3.
I20240506 15:13:08.399148 19974 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:08.399755 20060 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:08.399969 19934 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 3.
I20240506 15:13:08.400926 20060 raft_consensus.cc:686] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:08.401831 20060 consensus_queue.cc:237] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.409317 19757 catalog_manager.cc:5484] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 2 to 3, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 3 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:08.589757 20128 raft_consensus.cc:979] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:08.590168 20128 raft_consensus.cc:1067] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Signalling peer 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:08.591445 20007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:32970
I20240506 15:13:08.591945 20007 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:08.592209 20007 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:08.596375 20007 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.597841 20007 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:08.598713 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:08.598872 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:08.599195 19858 raft_consensus.cc:3032] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:08.599385 19934 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:08.599488 19858 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:08.600152 19858 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.601081 19858 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:08.603749 19934 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 2.
I20240506 15:13:08.604667 19975 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:08.605324 20060 raft_consensus.cc:2781] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:08.605284 19858 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 2.
I20240506 15:13:08.605916 20060 raft_consensus.cc:686] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:08.606746 20060 consensus_queue.cc:237] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:08.615442 19757 catalog_manager.cc:5484] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 1 to 2, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 2 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:08.881436 19934 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:08.882601 20060 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:08.899279 19858 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:08.900681 20060 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:09.015722 19858 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:09.016896 20060 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:09.024533 19934 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:09.025779 20152 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:09.363520 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:09.368486 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:09.368849 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:10.369634 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:10.374711 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:10.375092 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:11.377391 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:11.384003 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:11.384616 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:12.385452 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:12.390825 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:12.391259 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:13.419677 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:13.424538 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:13.424914 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240506 15:13:14.032559 19882 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea060 after lost signal to thread 19729
W20240506 15:13:14.033324 19882 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea740 after lost signal to thread 19792
W20240506 15:13:14.034122 19882 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc740 after lost signal to thread 19885
W20240506 15:13:14.034761 19882 debug-util.cc:398] Leaking SignalData structure 0x7b0800117160 after lost signal to thread 19960
W20240506 15:13:14.035367 19882 debug-util.cc:398] Leaking SignalData structure 0x7b08001dce60 after lost signal to thread 20034
I20240506 15:13:14.425696 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:14.430717 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:14.431241 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:15.432104 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:15.437233 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:15.437717 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:16.438611 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:16.443603 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:16.444027 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:17.444839 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:17.451134 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:17.451725 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:18.479399 19726 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240506 15:13:18.484400 19858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:51550
I20240506 15:13:18.485038 19858 raft_consensus.cc:595] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:18.491143 20009 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:32992
I20240506 15:13:18.491626 20009 raft_consensus.cc:595] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:18.496424 19858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:51550
I20240506 15:13:18.496927 19858 raft_consensus.cc:595] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:18.501453 19858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5565a41f820419081e56978575cad4a"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:51550
I20240506 15:13:18.501958 19858 raft_consensus.cc:595] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:18.506600 20009 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
mode: GRACEFUL
new_leader_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:32992
I20240506 15:13:18.507118 20009 raft_consensus.cc:595] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Received request to transfer leadership to TS fa025decf08f4c6689d5dd7b541f980c
I20240506 15:13:18.617785 20172 raft_consensus.cc:979] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.618208 20172 raft_consensus.cc:1067] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.619896 19935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33996
I20240506 15:13:18.620430 19935 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:18.620762 19935 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:18.626096 19935 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.627792 19935 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 4 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:18.628825 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:18.628986 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:18.629593 19858 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:18.629817 20009 raft_consensus.cc:3032] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:13:18.630169 20009 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:18.630879 20009 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.632054 20009 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:18.634826 19858 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 4.
I20240506 15:13:18.636171 19900 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:18.637113 20175 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:13:18.638033 20176 raft_consensus.cc:979] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.638515 20176 raft_consensus.cc:1067] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.638798 20175 raft_consensus.cc:686] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:18.639811 20175 consensus_queue.cc:237] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.640349 19935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33982
I20240506 15:13:18.640904 19935 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:18.641269 19935 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.644542 20009 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 4.
I20240506 15:13:18.649827 19935 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.652217 19935 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 3 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:18.652941 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:18.653553 19858 raft_consensus.cc:3032] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:18.653487 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:18.654007 19858 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:18.654207 20009 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.654825 19858 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.655517 20128 raft_consensus.cc:979] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.655941 20128 raft_consensus.cc:1067] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.656535 19858 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.657908 19935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a5565a41f820419081e56978575cad4a"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33982
I20240506 15:13:18.658572 19935 raft_consensus.cc:482] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:18.659251 19935 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.658407 19756 catalog_manager.cc:5484] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 3 to 4, leader changed from 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 4 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:18.661825 20009 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 3.
I20240506 15:13:18.663681 19858 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 3.
I20240506 15:13:18.664008 19899 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:18.666059 20175 raft_consensus.cc:2781] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:18.666603 20175 raft_consensus.cc:686] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:18.666692 19935 raft_consensus.cc:504] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.667364 20128 raft_consensus.cc:979] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.667873 20128 raft_consensus.cc:1067] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.668787 19935 leader_election.cc:288] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:18.667723 20175 consensus_queue.cc:237] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.670123 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:18.670290 19934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33982
I20240506 15:13:18.671077 20009 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.671166 19934 raft_consensus.cc:482] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:18.671700 19934 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.672464 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5565a41f820419081e56978575cad4a" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:18.673115 19861 raft_consensus.cc:3032] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:18.673434 19861 raft_consensus.cc:729] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:18.674808 19861 consensus_queue.cc:260] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.676640 19861 raft_consensus.cc:3037] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.679011 20009 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:18.680285 19899 leader_election.cc:302] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:18.681407 20178 raft_consensus.cc:2781] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:18.682202 20178 raft_consensus.cc:686] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:18.681380 19934 raft_consensus.cc:504] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.683106 20178 consensus_queue.cc:237] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.686386 19861 raft_consensus.cc:2454] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:18.695226 19934 leader_election.cc:288] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:18.695755 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:18.695755 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f217906cf442258a45bd798ac4ed03" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:18.696585 20009 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.696779 19861 raft_consensus.cc:3032] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:18.697319 19861 raft_consensus.cc:729] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:18.698554 19861 consensus_queue.cc:260] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.700621 19861 raft_consensus.cc:3037] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:18.702414 20009 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:18.703723 19899 leader_election.cc:302] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:18.704475 20178 raft_consensus.cc:2781] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:18.704898 20178 raft_consensus.cc:686] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:18.705773 20178 consensus_queue.cc:237] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.708801 19861 raft_consensus.cc:2454] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 2.
I20240506 15:13:18.711740 20172 raft_consensus.cc:979] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: : Instructing follower fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.712218 20172 raft_consensus.cc:1067] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Signalling peer fa025decf08f4c6689d5dd7b541f980c to start an election
I20240506 15:13:18.713575 19934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
from {username='slave'} at 127.0.0.1:33996
I20240506 15:13:18.713997 19934 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:18.714258 19934 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.714879 19756 catalog_manager.cc:5484] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 2 to 3, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 3 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:18.715978 19756 catalog_manager.cc:5484] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:18.717010 19756 catalog_manager.cc:5484] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 1 to 2, leader changed from 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 2 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:18.719345 19934 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.721942 19934 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 3 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:18.721992 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:18.722265 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "fa025decf08f4c6689d5dd7b541f980c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:18.722849 20009 raft_consensus.cc:3032] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:18.723343 20009 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:18.722849 19861 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.724180 20009 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.726039 20009 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:18.729032 19861 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 3.
I20240506 15:13:18.729887 19900 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:18.730496 20178 raft_consensus.cc:2781] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:18.730973 20178 raft_consensus.cc:686] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Becoming Leader. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:18.731949 20009 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa025decf08f4c6689d5dd7b541f980c in term 3.
I20240506 15:13:18.731951 20178 consensus_queue.cc:237] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:18.739954 19756 catalog_manager.cc:5484] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c reported cstate change: term changed from 2 to 3, leader changed from 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131) to fa025decf08f4c6689d5dd7b541f980c (127.19.67.130). New cstate: current_term: 3 leader_uuid: "fa025decf08f4c6689d5dd7b541f980c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:19.031003 20009 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:19.032228 20178 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:19.041781 19861 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:19.043291 20178 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:19.131522 19861 raft_consensus.cc:1261] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:19.132742 20178 consensus_queue.cc:1035] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:19.141845 20009 raft_consensus.cc:1261] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:19.143083 20175 consensus_queue.cc:1035] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:19.157133 20009 raft_consensus.cc:1261] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:19.158463 20178 consensus_queue.cc:1035] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:19.163872 19861 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:19.165426 20178 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:19.167752 19861 raft_consensus.cc:1261] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:19.169836 20175 consensus_queue.cc:1035] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:19.176110 19860 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:19.182499 20175 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:19.184150 20009 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:19.185541 20175 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:13:19.194345 20008 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Refusing update from remote peer fa025decf08f4c6689d5dd7b541f980c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:19.196094 20199 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:13:21.546059 19726 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240506 15:13:21.546969 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:21.554569 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:21.555339 19934 raft_consensus.cc:595] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Received request to transfer leadership to TS 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:21.560461 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:21.561703 19934 raft_consensus.cc:595] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Received request to transfer leadership to TS 4a2eb00bc6d8498e99de9079eec2f7de
I20240506 15:13:21.565999 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:21.566797 19934 raft_consensus.cc:595] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Received request to transfer leadership to TS 67088c3e2838499e85f0cf4aea0d12c8
I20240506 15:13:21.569370 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:21.569854 19934 raft_consensus.cc:595] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 LEADER]: Received request to transfer leadership to TS 67088c3e2838499e85f0cf4aea0d12c8
I20240506 15:13:21.572116 19934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
mode: GRACEFUL
new_leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:34010
I20240506 15:13:21.572700 19934 raft_consensus.cc:595] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Received request to transfer leadership to TS 67088c3e2838499e85f0cf4aea0d12c8
I20240506 15:13:21.573730 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:13:21.574193 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:21.576097 20179 raft_consensus.cc:979] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.577533 20175 raft_consensus.cc:1067] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Signalling peer 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.580327 19860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:51548
I20240506 15:13:21.580610 20179 raft_consensus.cc:979] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.580891 19860 raft_consensus.cc:482] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:21.581175 20204 raft_consensus.cc:1067] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 LEADER]: Signalling peer 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.581323 19860 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.585498 19861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:51548
I20240506 15:13:21.586087 19861 raft_consensus.cc:482] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:21.586428 19861 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:21.673185 19861 raft_consensus.cc:504] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.672875 19860 raft_consensus.cc:504] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.677330 19860 leader_election.cc:288] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 4 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:21.679040 20008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:21.679400 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:21.679704 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e682ccc2da5640e289ee1efacf2b47f5" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:21.679982 20008 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.680176 19934 raft_consensus.cc:3032] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:13:21.680434 19935 raft_consensus.cc:3032] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:13:21.680639 19861 leader_election.cc:288] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 5 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:21.680868 19935 raft_consensus.cc:729] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:21.682204 19935 consensus_queue.cc:260] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.683792 20204 raft_consensus.cc:979] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.683792 19935 raft_consensus.cc:3037] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.683900 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49add7ebe9ce4c9a9b607d9298478113" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:21.684844 20009 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:21.680644 19934 raft_consensus.cc:729] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:21.686175 19934 consensus_queue.cc:260] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.684422 20179 raft_consensus.cc:1067] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Signalling peer 67088c3e2838499e85f0cf4aea0d12c8 to start an election
I20240506 15:13:21.688267 19934 raft_consensus.cc:3037] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:21.690147 19861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61"
dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
from {username='slave'} at 127.0.0.1:51548
I20240506 15:13:21.690771 19861 raft_consensus.cc:482] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:21.691180 19861 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.738974 20204 raft_consensus.cc:979] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:21.739472 20179 raft_consensus.cc:1067] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Signalling peer 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:21.741154 20010 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34b195d05f8246c082221976baa6842f"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:32986
I20240506 15:13:21.741601 20010 raft_consensus.cc:482] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:21.741855 20010 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.774322 20008 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 4.
I20240506 15:13:21.774457 19935 raft_consensus.cc:2454] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 4.
I20240506 15:13:21.775003 19934 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 5.
I20240506 15:13:21.774510 19861 raft_consensus.cc:504] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.776357 20009 raft_consensus.cc:2454] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 5.
I20240506 15:13:21.776643 19824 leader_election.cc:302] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:21.778322 19861 leader_election.cc:288] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 3 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131:34721)
I20240506 15:13:21.778851 19826 leader_election.cc:302] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67088c3e2838499e85f0cf4aea0d12c8, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:21.779714 20205 raft_consensus.cc:2781] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:13:21.781905 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:21.782714 19935 raft_consensus.cc:3032] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:21.783138 19935 raft_consensus.cc:729] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:21.784828 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af3545a5d74d0e99626672ab116d61" candidate_uuid: "67088c3e2838499e85f0cf4aea0d12c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
I20240506 15:13:21.788473 19935 consensus_queue.cc:260] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.790836 20009 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.790967 19935 raft_consensus.cc:3037] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.799083 20205 raft_consensus.cc:686] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:21.799566 20207 raft_consensus.cc:2781] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:13:21.801100 20205 consensus_queue.cc:237] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.804773 20207 raft_consensus.cc:686] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 5 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:21.806236 20207 consensus_queue.cc:237] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.808748 20179 raft_consensus.cc:979] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: : Instructing follower 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:21.809355 20179 raft_consensus.cc:1067] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Signalling peer 4a2eb00bc6d8498e99de9079eec2f7de to start an election
I20240506 15:13:21.811662 20008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195"
dest_uuid: "4a2eb00bc6d8498e99de9079eec2f7de"
from {username='slave'} at 127.0.0.1:32986
I20240506 15:13:21.812482 20008 raft_consensus.cc:482] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:21.812903 20008 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.813654 19756 catalog_manager.cc:5484] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 3 to 4, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 4 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:21.832649 19756 catalog_manager.cc:5484] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 4 to 5, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 5 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:21.870307 20009 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 3.
I20240506 15:13:21.870532 20010 raft_consensus.cc:504] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.871767 19824 leader_election.cc:302] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:21.872797 20207 raft_consensus.cc:2781] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:21.873245 20207 raft_consensus.cc:686] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 LEADER]: Becoming Leader. State: Replica: 67088c3e2838499e85f0cf4aea0d12c8, State: Running, Role: LEADER
I20240506 15:13:21.874501 20010 leader_election.cc:288] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 4 election: Requested vote from peers fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639), 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149)
I20240506 15:13:21.874171 20207 consensus_queue.cc:237] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.870307 19935 raft_consensus.cc:2454] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67088c3e2838499e85f0cf4aea0d12c8 in term 3.
I20240506 15:13:21.875759 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:21.876663 19934 raft_consensus.cc:3032] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:13:21.877008 19934 raft_consensus.cc:729] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:21.876688 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b195d05f8246c082221976baa6842f" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:21.877885 19861 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.877686 19934 consensus_queue.cc:260] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.879011 19934 raft_consensus.cc:3037] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:21.883836 19756 catalog_manager.cc:5484] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 reported cstate change: term changed from 2 to 3, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129). New cstate: current_term: 3 leader_uuid: "67088c3e2838499e85f0cf4aea0d12c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:21.966718 19934 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 4.
I20240506 15:13:21.966766 19861 raft_consensus.cc:2454] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 4.
I20240506 15:13:21.966837 20008 raft_consensus.cc:504] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.969038 19975 leader_election.cc:302] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, fa025decf08f4c6689d5dd7b541f980c; no voters:
I20240506 15:13:21.969723 20008 leader_election.cc:288] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 3 election: Requested vote from peers 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149), fa025decf08f4c6689d5dd7b541f980c (127.19.67.130:37639)
I20240506 15:13:21.970695 20208 raft_consensus.cc:2781] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:13:21.971195 20208 raft_consensus.cc:686] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:21.971349 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa025decf08f4c6689d5dd7b541f980c"
I20240506 15:13:21.971717 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc75e349edd4cba9b6222666a2b7195" candidate_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67088c3e2838499e85f0cf4aea0d12c8"
I20240506 15:13:21.971993 19934 raft_consensus.cc:3032] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:21.972404 19861 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.972481 19934 raft_consensus.cc:729] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa025decf08f4c6689d5dd7b541f980c, State: Running, Role: LEADER
I20240506 15:13:21.972071 20208 consensus_queue.cc:237] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.973277 19934 consensus_queue.cc:260] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:21.974241 19934 raft_consensus.cc:3037] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:21.980958 19756 catalog_manager.cc:5484] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 3 to 4, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 4 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:22.068097 19861 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 3.
I20240506 15:13:22.068100 19934 raft_consensus.cc:2454] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a2eb00bc6d8498e99de9079eec2f7de in term 3.
I20240506 15:13:22.069551 19974 leader_election.cc:302] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2eb00bc6d8498e99de9079eec2f7de, 67088c3e2838499e85f0cf4aea0d12c8; no voters:
I20240506 15:13:22.071067 20208 raft_consensus.cc:2781] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:22.072069 20208 raft_consensus.cc:686] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Becoming Leader. State: Replica: 4a2eb00bc6d8498e99de9079eec2f7de, State: Running, Role: LEADER
I20240506 15:13:22.073411 20208 consensus_queue.cc:237] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } }
I20240506 15:13:22.082152 19756 catalog_manager.cc:5484] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de reported cstate change: term changed from 2 to 3, leader changed from fa025decf08f4c6689d5dd7b541f980c (127.19.67.130) to 4a2eb00bc6d8498e99de9079eec2f7de (127.19.67.131). New cstate: current_term: 3 leader_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:22.192893 20008 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 5 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:13:22.194005 20207 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:13:22.204797 19934 raft_consensus.cc:1261] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 5 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:13:22.206218 20205 consensus_queue.cc:1035] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:13:22.220458 19934 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:22.221616 20205 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:22.231072 20008 raft_consensus.cc:1261] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:22.233412 20205 consensus_queue.cc:1035] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:13:22.412034 19934 raft_consensus.cc:1261] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:22.413470 20205 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:22.425447 20008 raft_consensus.cc:1261] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Refusing update from remote peer 67088c3e2838499e85f0cf4aea0d12c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:22.426993 20205 consensus_queue.cc:1035] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2eb00bc6d8498e99de9079eec2f7de" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:22.510368 19861 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:22.511555 20208 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:22.520959 19934 raft_consensus.cc:1261] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:22.523039 20208 consensus_queue.cc:1035] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:22.548668 19861 raft_consensus.cc:1261] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:22.550123 20208 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67088c3e2838499e85f0cf4aea0d12c8" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:13:22.564165 19934 raft_consensus.cc:1261] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Refusing update from remote peer 4a2eb00bc6d8498e99de9079eec2f7de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:22.568980 20208 consensus_queue.cc:1035] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa025decf08f4c6689d5dd7b541f980c" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:22.575762 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:22.583981 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:22.584453 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:23.585172 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:23.590368 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:23.590756 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:24.591400 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:24.596151 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:24.596621 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:25.597389 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:25.602279 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:25.602710 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:26.603662 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:26.610020 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:26.610492 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:27.611253 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:27.616657 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:27.617054 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:28.618080 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:28.623174 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:28.623639 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:29.624701 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:29.629776 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:29.630280 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:30.631191 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:30.636969 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:30.637403 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:31.665122 19726 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240506 15:13:31.665870 19726 tablet_server.cc:178] TabletServer@127.19.67.129:0 shutting down...
I20240506 15:13:31.702150 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:13:31.702927 19726 tablet_replica.cc:330] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.703640 19726 raft_consensus.cc:2229] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 LEADER]: Raft consensus shutting down.
I20240506 15:13:31.704949 19726 raft_consensus.cc:2258] T 35af3545a5d74d0e99626672ab116d61 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.707440 19726 tablet_replica.cc:330] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.708091 19726 raft_consensus.cc:2229] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.708582 19726 raft_consensus.cc:2258] T 135f3f56fc8c4bdf8821262d9d210349 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.710407 19726 tablet_replica.cc:330] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.710935 19726 raft_consensus.cc:2229] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.711407 19726 raft_consensus.cc:2258] T 34b195d05f8246c082221976baa6842f P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.713495 19726 tablet_replica.cc:330] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.714013 19726 raft_consensus.cc:2229] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.714455 19726 raft_consensus.cc:2258] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.716279 19726 tablet_replica.cc:330] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.716766 19726 raft_consensus.cc:2229] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.717164 19726 raft_consensus.cc:2258] T 2dc75e349edd4cba9b6222666a2b7195 P 67088c3e2838499e85f0cf4aea0d12c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.718844 19726 tablet_replica.cc:330] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.719312 19726 raft_consensus.cc:2229] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 LEADER]: Raft consensus shutting down.
I20240506 15:13:31.720927 19726 raft_consensus.cc:2258] T e682ccc2da5640e289ee1efacf2b47f5 P 67088c3e2838499e85f0cf4aea0d12c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.722832 19726 tablet_replica.cc:330] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.723315 19726 raft_consensus.cc:2229] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.723739 19726 raft_consensus.cc:2258] T 63f217906cf442258a45bd798ac4ed03 P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.725673 19726 tablet_replica.cc:330] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.726445 19726 raft_consensus.cc:2229] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240506 15:13:31.726713 19974 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240506 15:13:31.726992 19726 raft_consensus.cc:2258] T a5565a41f820419081e56978575cad4a P 67088c3e2838499e85f0cf4aea0d12c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.729243 19726 tablet_replica.cc:330] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8: stopping tablet replica
I20240506 15:13:31.729799 19726 raft_consensus.cc:2229] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 5 LEADER]: Raft consensus shutting down.
W20240506 15:13:31.730206 19974 consensus_peers.cc:485] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de -> Peer 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149): Couldn't send request to peer 67088c3e2838499e85f0cf4aea0d12c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:13:31.730976 19726 raft_consensus.cc:2258] T 49add7ebe9ce4c9a9b607d9298478113 P 67088c3e2838499e85f0cf4aea0d12c8 [term 5 FOLLOWER]: Raft consensus is shut down!
W20240506 15:13:31.873397 19900 consensus_peers.cc:485] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c -> Peer 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149): Couldn't send request to peer 67088c3e2838499e85f0cf4aea0d12c8. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.129:44149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:13:31.891546 19726 tablet_server.cc:194] TabletServer@127.19.67.129:0 shutdown complete.
I20240506 15:13:31.909826 19726 tablet_server.cc:178] TabletServer@127.19.67.130:0 shutting down...
I20240506 15:13:31.948184 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:13:31.949072 19726 tablet_replica.cc:330] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.949792 19726 raft_consensus.cc:2229] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.950343 19726 raft_consensus.cc:2258] T 35af3545a5d74d0e99626672ab116d61 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.952513 19726 tablet_replica.cc:330] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.953190 19726 raft_consensus.cc:2229] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.954010 19726 raft_consensus.cc:2258] T 14c3ff62d52c4e2a82761c3458e2c8d3 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.956652 19726 tablet_replica.cc:330] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
W20240506 15:13:31.957298 19900 consensus_peers.cc:485] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c -> Peer 67088c3e2838499e85f0cf4aea0d12c8 (127.19.67.129:44149): Couldn't send request to peer 67088c3e2838499e85f0cf4aea0d12c8. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.129:44149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:13:31.957656 19726 raft_consensus.cc:2229] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.958199 19726 raft_consensus.cc:2258] T 2dc75e349edd4cba9b6222666a2b7195 P fa025decf08f4c6689d5dd7b541f980c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.960053 19726 tablet_replica.cc:330] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.960639 19726 raft_consensus.cc:2229] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.961117 19726 raft_consensus.cc:2258] T 34b195d05f8246c082221976baa6842f P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.962884 19726 tablet_replica.cc:330] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.963415 19726 raft_consensus.cc:2229] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:13:31.964567 19726 raft_consensus.cc:2258] T a5565a41f820419081e56978575cad4a P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.966599 19726 tablet_replica.cc:330] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.967278 19726 raft_consensus.cc:2229] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 5 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.967762 19726 raft_consensus.cc:2258] T 49add7ebe9ce4c9a9b607d9298478113 P fa025decf08f4c6689d5dd7b541f980c [term 5 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.969604 19726 tablet_replica.cc:330] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.970111 19726 raft_consensus.cc:2229] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:13:31.970908 19726 raft_consensus.cc:2258] T 135f3f56fc8c4bdf8821262d9d210349 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.972729 19726 tablet_replica.cc:330] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.973203 19726 raft_consensus.cc:2229] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:31.973696 19726 raft_consensus.cc:2258] T e682ccc2da5640e289ee1efacf2b47f5 P fa025decf08f4c6689d5dd7b541f980c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:31.975683 19726 tablet_replica.cc:330] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c: stopping tablet replica
I20240506 15:13:31.976155 19726 raft_consensus.cc:2229] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:13:31.976995 19726 raft_consensus.cc:2258] T 63f217906cf442258a45bd798ac4ed03 P fa025decf08f4c6689d5dd7b541f980c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.009245 19726 tablet_server.cc:194] TabletServer@127.19.67.130:0 shutdown complete.
I20240506 15:13:32.029588 19726 tablet_server.cc:178] TabletServer@127.19.67.131:0 shutting down...
I20240506 15:13:32.071687 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:13:32.072446 19726 tablet_replica.cc:330] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.073405 19726 raft_consensus.cc:2229] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 LEADER]: Raft consensus shutting down.
I20240506 15:13:32.074664 19726 raft_consensus.cc:2258] T 34b195d05f8246c082221976baa6842f P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.076685 19726 tablet_replica.cc:330] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.077195 19726 raft_consensus.cc:2229] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.077711 19726 raft_consensus.cc:2258] T 35af3545a5d74d0e99626672ab116d61 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.079471 19726 tablet_replica.cc:330] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.079974 19726 raft_consensus.cc:2229] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 LEADER]: Raft consensus shutting down.
I20240506 15:13:32.080838 19726 raft_consensus.cc:2258] T 2dc75e349edd4cba9b6222666a2b7195 P 4a2eb00bc6d8498e99de9079eec2f7de [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.082679 19726 tablet_replica.cc:330] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.083282 19726 raft_consensus.cc:2229] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:13:32.084232 19726 raft_consensus.cc:2258] T 14c3ff62d52c4e2a82761c3458e2c8d3 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.086373 19726 tablet_replica.cc:330] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.087453 19726 raft_consensus.cc:2229] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.087934 19726 raft_consensus.cc:2258] T 63f217906cf442258a45bd798ac4ed03 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.090318 19726 tablet_replica.cc:330] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.090883 19726 raft_consensus.cc:2229] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.091346 19726 raft_consensus.cc:2258] T e682ccc2da5640e289ee1efacf2b47f5 P 4a2eb00bc6d8498e99de9079eec2f7de [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.093241 19726 tablet_replica.cc:330] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.093729 19726 raft_consensus.cc:2229] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.094141 19726 raft_consensus.cc:2258] T a5565a41f820419081e56978575cad4a P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.095746 19726 tablet_replica.cc:330] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.096195 19726 raft_consensus.cc:2229] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 5 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.096678 19726 raft_consensus.cc:2258] T 49add7ebe9ce4c9a9b607d9298478113 P 4a2eb00bc6d8498e99de9079eec2f7de [term 5 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.098192 19726 tablet_replica.cc:330] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de: stopping tablet replica
I20240506 15:13:32.098642 19726 raft_consensus.cc:2229] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:13:32.099268 19726 raft_consensus.cc:2258] T 135f3f56fc8c4bdf8821262d9d210349 P 4a2eb00bc6d8498e99de9079eec2f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.134164 19726 tablet_server.cc:194] TabletServer@127.19.67.131:0 shutdown complete.
I20240506 15:13:32.155045 19726 master.cc:537] Master@127.19.67.190:33405 shutting down...
I20240506 15:13:32.369518 19726 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:13:32.370170 19726 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:13:32.370481 19726 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f7a0ea266964d9a9dcb131bcac9ff7a: stopping tablet replica
I20240506 15:13:32.389582 19726 master.cc:552] Master@127.19.67.190:33405 shutdown complete.
4a2eb00bc6d8498e99de9079eec2f7de 1
67088c3e2838499e85f0cf4aea0d12c8 4
fa025decf08f4c6689d5dd7b541f980c 4
4a2eb00bc6d8498e99de9079eec2f7de 3
67088c3e2838499e85f0cf4aea0d12c8 3
fa025decf08f4c6689d5dd7b541f980c 3
4a2eb00bc6d8498e99de9079eec2f7de 1
67088c3e2838499e85f0cf4aea0d12c8 0
fa025decf08f4c6689d5dd7b541f980c 8
4a2eb00bc6d8498e99de9079eec2f7de 3
67088c3e2838499e85f0cf4aea0d12c8 3
fa025decf08f4c6689d5dd7b541f980c 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31104 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240506 15:13:32.435856 19726 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.67.190:35243
I20240506 15:13:32.436978 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:32.442832 20245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.444517 20246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.444737 19726 server_base.cc:1034] running on GCE node
W20240506 15:13:32.445787 20248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.446656 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:32.446851 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:32.446977 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008412446966 us; error 0 us; skew 500 ppm
I20240506 15:13:32.447448 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:32.450320 19726 webserver.cc:460] Webserver started at http://127.19.67.190:38043/ using document root <none> and password file <none>
I20240506 15:13:32.450798 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:32.450959 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:32.451189 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:32.452662 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/master-0-root/instance:
uuid: "1d4b7e87dc814e58a45f3a243f6d9657"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.457893 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 15:13:32.461369 20253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.462347 19726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240506 15:13:32.462620 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/master-0-root
uuid: "1d4b7e87dc814e58a45f3a243f6d9657"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.462872 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:32.473366 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:32.474550 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:32.514256 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.190:35243
I20240506 15:13:32.514344 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.190:35243 every 8 connection(s)
I20240506 15:13:32.518431 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:32.528412 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657: Bootstrap starting.
I20240506 15:13:32.532564 20305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:32.536818 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657: No bootstrap required, opened a new log
I20240506 15:13:32.538772 20305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER }
I20240506 15:13:32.539227 20305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:32.539465 20305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4b7e87dc814e58a45f3a243f6d9657, State: Initialized, Role: FOLLOWER
I20240506 15:13:32.540036 20305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER }
I20240506 15:13:32.540549 20305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:13:32.540800 20305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:13:32.541134 20305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:32.545130 20305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER }
I20240506 15:13:32.545789 20305 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4b7e87dc814e58a45f3a243f6d9657; no voters:
I20240506 15:13:32.546974 20305 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:13:32.547403 20308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:32.548878 20308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 1 LEADER]: Becoming Leader. State: Replica: 1d4b7e87dc814e58a45f3a243f6d9657, State: Running, Role: LEADER
I20240506 15:13:32.549525 20308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER }
I20240506 15:13:32.550101 20305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:13:32.556576 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER } }
I20240506 15:13:32.557344 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [sys.catalog]: This master's current role is: LEADER
I20240506 15:13:32.561369 20310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4b7e87dc814e58a45f3a243f6d9657. Latest consensus state: current_term: 1 leader_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b7e87dc814e58a45f3a243f6d9657" member_type: VOTER } }
I20240506 15:13:32.563166 20310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [sys.catalog]: This master's current role is: LEADER
I20240506 15:13:32.570555 20315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:13:32.579226 20315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:13:32.580293 19726 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240506 15:13:32.588325 20315 catalog_manager.cc:1347] Generated new cluster ID: facf347e438a4f5c8210fe17f6ef2835
I20240506 15:13:32.588657 20315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:13:32.632664 20315 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:13:32.633915 20315 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:13:32.648983 20315 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657: Generated new TSK 0
I20240506 15:13:32.649623 20315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:13:32.711195 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:32.716930 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.719198 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.719851 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.721549 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:32.722445 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:32.722640 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:32.722796 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008412722783 us; error 0 us; skew 500 ppm
I20240506 15:13:32.723348 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:32.725943 19726 webserver.cc:460] Webserver started at http://127.19.67.129:36259/ using document root <none> and password file <none>
I20240506 15:13:32.726454 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:32.726631 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:32.726900 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:32.727993 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-0-root/instance:
uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.732791 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 15:13:32.736095 20334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.736917 19726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 15:13:32.737206 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.737473 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:32.756789 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:32.758000 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:32.759471 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:32.761847 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:32.762027 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.762205 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:32.762394 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.804639 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.129:34331
I20240506 15:13:32.804728 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.129:34331 every 8 connection(s)
I20240506 15:13:32.810120 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:32.817998 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.821521 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.822311 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.823920 20397 heartbeater.cc:347] Connected to a master server at 127.19.67.190:35243
I20240506 15:13:32.824074 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:32.824395 20397 heartbeater.cc:464] Registering TS with master...
I20240506 15:13:32.825455 20397 heartbeater.cc:511] Master 127.19.67.190:35243 requested a full tablet report, sending...
I20240506 15:13:32.825883 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:32.826181 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:32.826411 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008412826392 us; error 0 us; skew 500 ppm
I20240506 15:13:32.827428 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:32.828092 20270 ts_manager.cc:194] Registered new tserver with Master: 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331)
I20240506 15:13:32.830324 19726 webserver.cc:460] Webserver started at http://127.19.67.130:44793/ using document root <none> and password file <none>
I20240506 15:13:32.830600 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47322
I20240506 15:13:32.831017 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:32.831292 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:32.831648 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:32.832942 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-1-root/instance:
uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.837400 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240506 15:13:32.841256 20409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.842060 19726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240506 15:13:32.842348 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.842625 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:32.863183 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:32.864568 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:32.865993 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:32.868422 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:32.868638 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.868871 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:32.869032 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.913339 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.130:32899
I20240506 15:13:32.913522 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.130:32899 every 8 connection(s)
I20240506 15:13:32.918829 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:13:32.926519 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:13:32.930363 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.930732 19726 server_base.cc:1034] running on GCE node
I20240506 15:13:32.931339 20472 heartbeater.cc:347] Connected to a master server at 127.19.67.190:35243
I20240506 15:13:32.931769 20472 heartbeater.cc:464] Registering TS with master...
W20240506 15:13:32.932166 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:13:32.933007 20472 heartbeater.cc:511] Master 127.19.67.190:35243 requested a full tablet report, sending...
I20240506 15:13:32.933156 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:13:32.933424 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:13:32.933578 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008412933560 us; error 0 us; skew 500 ppm
I20240506 15:13:32.934289 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:13:32.934955 20270 ts_manager.cc:194] Registered new tserver with Master: c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:32.936235 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47338
I20240506 15:13:32.937273 19726 webserver.cc:460] Webserver started at http://127.19.67.131:39119/ using document root <none> and password file <none>
I20240506 15:13:32.937842 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:13:32.938040 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:13:32.938297 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:13:32.939554 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-2-root/instance:
uuid: "7a0a151b64f54c71a48043fe8c832de8"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.944401 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240506 15:13:32.947850 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.948827 19726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 15:13:32.949124 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
uuid: "7a0a151b64f54c71a48043fe8c832de8"
format_stamp: "Formatted at 2024-05-06 15:13:32 on dist-test-slave-8gjm"
I20240506 15:13:32.949412 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715008381158361-19726-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:13:32.959894 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:13:32.961309 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:13:32.962582 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:13:32.965021 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:13:32.965224 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:32.965454 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:13:32.965606 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:13:33.004979 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.131:42587
I20240506 15:13:33.005076 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.131:42587 every 8 connection(s)
I20240506 15:13:33.018427 20546 heartbeater.cc:347] Connected to a master server at 127.19.67.190:35243
I20240506 15:13:33.018798 20546 heartbeater.cc:464] Registering TS with master...
I20240506 15:13:33.019516 20546 heartbeater.cc:511] Master 127.19.67.190:35243 requested a full tablet report, sending...
I20240506 15:13:33.021337 20270 ts_manager.cc:194] Registered new tserver with Master: 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.021761 19726 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013433705s
I20240506 15:13:33.022750 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47348
I20240506 15:13:33.024214 19726 test_util.cc:268] Using random seed: -2093039681
I20240506 15:13:33.047003 20270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47362:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:13:33.049899 20270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:13:33.122556 20362 tablet_service.cc:1467] Processing CreateTablet for tablet da6e40f376fc4c8f8ff453ea1830a594 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 15:13:33.123193 20361 tablet_service.cc:1467] Processing CreateTablet for tablet d1fea309275b4b52b577085109a48795 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 15:13:33.124296 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6e40f376fc4c8f8ff453ea1830a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.127039 20360 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 15:13:33.128512 20360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.133615 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fea309275b4b52b577085109a48795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.134117 20359 tablet_service.cc:1467] Processing CreateTablet for tablet eda181b266b74a7388b5a86923942cb2 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 15:13:33.137496 20357 tablet_service.cc:1467] Processing CreateTablet for tablet e6c21597f811461ba94aac038acf0a0b (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 15:13:33.138984 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c21597f811461ba94aac038acf0a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.141551 20356 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6243c28ccf4b60b4f9ad7219616621 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 15:13:33.142697 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda181b266b74a7388b5a86923942cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.143950 20436 tablet_service.cc:1467] Processing CreateTablet for tablet d1fea309275b4b52b577085109a48795 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 15:13:33.144084 20355 tablet_service.cc:1467] Processing CreateTablet for tablet ac3f567b0fef40059673b90d4899f471 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 15:13:33.145429 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fea309275b4b52b577085109a48795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.145629 20355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3f567b0fef40059673b90d4899f471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.145939 20437 tablet_service.cc:1467] Processing CreateTablet for tablet da6e40f376fc4c8f8ff453ea1830a594 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 15:13:33.148092 20432 tablet_service.cc:1467] Processing CreateTablet for tablet e6c21597f811461ba94aac038acf0a0b (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 15:13:33.149472 20435 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 15:13:33.150085 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6243c28ccf4b60b4f9ad7219616621 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 15:13:33.151489 20434 tablet_service.cc:1467] Processing CreateTablet for tablet eda181b266b74a7388b5a86923942cb2 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 15:13:33.152074 20437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6e40f376fc4c8f8ff453ea1830a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.141778 20358 tablet_service.cc:1467] Processing CreateTablet for tablet 5d70f8fe2e65455895c3f33d2fc84b91 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 15:13:33.153755 20433 tablet_service.cc:1467] Processing CreateTablet for tablet 5d70f8fe2e65455895c3f33d2fc84b91 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 15:13:33.154302 20356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6243c28ccf4b60b4f9ad7219616621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.154597 20434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda181b266b74a7388b5a86923942cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.154956 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d70f8fe2e65455895c3f33d2fc84b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.155221 20433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d70f8fe2e65455895c3f33d2fc84b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.156185 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6243c28ccf4b60b4f9ad7219616621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.160959 20435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.164870 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c21597f811461ba94aac038acf0a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.174296 20511 tablet_service.cc:1467] Processing CreateTablet for tablet da6e40f376fc4c8f8ff453ea1830a594 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 15:13:33.175705 20511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6e40f376fc4c8f8ff453ea1830a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.177484 20510 tablet_service.cc:1467] Processing CreateTablet for tablet d1fea309275b4b52b577085109a48795 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 15:13:33.178844 20510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fea309275b4b52b577085109a48795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.181525 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 15:13:33.182854 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4dc049405b4c75a4e8cf154ae8f7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.185644 20508 tablet_service.cc:1467] Processing CreateTablet for tablet eda181b266b74a7388b5a86923942cb2 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 15:13:33.187510 20508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda181b266b74a7388b5a86923942cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.189935 20507 tablet_service.cc:1467] Processing CreateTablet for tablet 5d70f8fe2e65455895c3f33d2fc84b91 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 15:13:33.191816 20507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d70f8fe2e65455895c3f33d2fc84b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.193918 20506 tablet_service.cc:1467] Processing CreateTablet for tablet e6c21597f811461ba94aac038acf0a0b (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 15:13:33.197227 20505 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6243c28ccf4b60b4f9ad7219616621 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 15:13:33.204057 20430 tablet_service.cc:1467] Processing CreateTablet for tablet ac3f567b0fef40059673b90d4899f471 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 15:13:33.205546 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3f567b0fef40059673b90d4899f471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.208829 20505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6243c28ccf4b60b4f9ad7219616621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.209955 20361 tablet_service.cc:1467] Processing CreateTablet for tablet a07f94eb5f7a4502a4ac643e466d4cf9 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 15:13:33.211295 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f94eb5f7a4502a4ac643e466d4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.212589 20506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c21597f811461ba94aac038acf0a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.217564 20504 tablet_service.cc:1467] Processing CreateTablet for tablet ac3f567b0fef40059673b90d4899f471 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 15:13:33.219013 20504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3f567b0fef40059673b90d4899f471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.243245 20561 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.276852 20503 tablet_service.cc:1467] Processing CreateTablet for tablet a07f94eb5f7a4502a4ac643e466d4cf9 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 15:13:33.278179 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f94eb5f7a4502a4ac643e466d4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.281510 20358 tablet_service.cc:1467] Processing CreateTablet for tablet 7e41659303f74c5084ff0a5d2ae0ac04 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 15:13:33.282719 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e41659303f74c5084ff0a5d2ae0ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.289125 20434 tablet_service.cc:1467] Processing CreateTablet for tablet a07f94eb5f7a4502a4ac643e466d4cf9 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 15:13:33.290431 20434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f94eb5f7a4502a4ac643e466d4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.297288 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 7e41659303f74c5084ff0a5d2ae0ac04 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 15:13:33.297741 20502 tablet_service.cc:1467] Processing CreateTablet for tablet 7e41659303f74c5084ff0a5d2ae0ac04 (DEFAULT_TABLE table=test-workload [id=b32df3e35da1458a889616a455df7c98]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 15:13:33.298666 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e41659303f74c5084ff0a5d2ae0ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.298929 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e41659303f74c5084ff0a5d2ae0ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:13:33.301430 20561 tablet_bootstrap.cc:654] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.324865 20563 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.331831 20561 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.332369 20561 ts_tablet_manager.cc:1367] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.005s
I20240506 15:13:33.334631 20563 tablet_bootstrap.cc:654] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.334398 20561 raft_consensus.cc:348] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.335111 20561 raft_consensus.cc:374] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.335263 20562 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.335470 20561 raft_consensus.cc:729] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.336310 20561 consensus_queue.cc:260] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.344190 20397 heartbeater.cc:503] Master 127.19.67.190:35243 was elected leader, sending a full tablet report...
I20240506 15:13:33.345736 20561 ts_tablet_manager.cc:1398] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20240506 15:13:33.347069 20562 tablet_bootstrap.cc:654] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.351104 20561 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.352190 20563 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.352761 20563 ts_tablet_manager.cc:1367] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.019s
I20240506 15:13:33.355921 20563 raft_consensus.cc:348] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.356710 20563 raft_consensus.cc:374] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.357014 20562 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.357017 20563 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.357391 20561 tablet_bootstrap.cc:654] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.357563 20562 ts_tablet_manager.cc:1367] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.002s
I20240506 15:13:33.358039 20563 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.360765 20562 raft_consensus.cc:348] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.361846 20562 raft_consensus.cc:374] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.362231 20562 raft_consensus.cc:729] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.363332 20562 consensus_queue.cc:260] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.366925 20546 heartbeater.cc:503] Master 127.19.67.190:35243 was elected leader, sending a full tablet report...
I20240506 15:13:33.368455 20563 ts_tablet_manager.cc:1398] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20240506 15:13:33.369983 20563 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.379567 20562 ts_tablet_manager.cc:1398] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.022s user 0.007s sys 0.014s
I20240506 15:13:33.380312 20561 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.380844 20561 ts_tablet_manager.cc:1367] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20240506 15:13:33.382033 20472 heartbeater.cc:503] Master 127.19.67.190:35243 was elected leader, sending a full tablet report...
I20240506 15:13:33.382422 20562 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.383327 20561 raft_consensus.cc:348] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.384094 20561 raft_consensus.cc:374] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.384425 20561 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.385125 20561 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.386929 20563 tablet_bootstrap.cc:654] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.387511 20561 ts_tablet_manager.cc:1398] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240506 15:13:33.388470 20561 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.388746 20562 tablet_bootstrap.cc:654] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.395925 20561 tablet_bootstrap.cc:654] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.409969 20561 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.410645 20561 ts_tablet_manager.cc:1367] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20240506 15:13:33.414238 20561 raft_consensus.cc:348] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.415347 20561 raft_consensus.cc:374] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.415737 20561 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.417032 20561 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.420060 20566 raft_consensus.cc:482] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.420950 20566 raft_consensus.cc:504] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.429936 20566 leader_election.cc:288] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.436005 20561 ts_tablet_manager.cc:1398] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.025s user 0.011s sys 0.000s
I20240506 15:13:33.441004 20561 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.445279 20566 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.447639 20566 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.450481 20566 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.461745 20561 tablet_bootstrap.cc:654] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.472121 20563 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.472761 20563 ts_tablet_manager.cc:1367] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.103s user 0.058s sys 0.033s
I20240506 15:13:33.476387 20563 raft_consensus.cc:348] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.477406 20563 raft_consensus.cc:374] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.477793 20563 raft_consensus.cc:729] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.478581 20561 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.478647 20563 consensus_queue.cc:260] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.479477 20561 ts_tablet_manager.cc:1367] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.007s
I20240506 15:13:33.482311 20563 ts_tablet_manager.cc:1398] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240506 15:13:33.483258 20563 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.482823 20561 raft_consensus.cc:348] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.483625 20561 raft_consensus.cc:374] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.483928 20561 raft_consensus.cc:729] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.484912 20561 consensus_queue.cc:260] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.489601 20563 tablet_bootstrap.cc:654] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.489640 20561 ts_tablet_manager.cc:1398] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240506 15:13:33.490826 20561 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.491524 20568 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.492017 20568 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.498183 20568 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.498250 20566 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.498962 20566 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.501657 20566 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.504889 20561 tablet_bootstrap.cc:654] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.511389 20562 tablet_bootstrap.cc:492] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.511941 20562 ts_tablet_manager.cc:1367] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.077s
I20240506 15:13:33.514724 20566 raft_consensus.cc:482] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.514791 20562 raft_consensus.cc:348] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.515903 20562 raft_consensus.cc:374] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.515621 20566 raft_consensus.cc:504] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.516373 20562 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.517144 20562 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.518545 20566 leader_election.cc:288] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.519731 20562 ts_tablet_manager.cc:1398] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240506 15:13:33.520756 20562 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.522430 20561 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.523204 20561 ts_tablet_manager.cc:1367] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.001s
I20240506 15:13:33.527184 20561 raft_consensus.cc:348] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.528395 20561 raft_consensus.cc:374] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.528859 20561 raft_consensus.cc:729] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.529361 20562 tablet_bootstrap.cc:654] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.530040 20561 consensus_queue.cc:260] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.534188 20561 ts_tablet_manager.cc:1398] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240506 15:13:33.534969 20561 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.535084 20563 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.535581 20563 ts_tablet_manager.cc:1367] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.026s
I20240506 15:13:33.538372 20563 raft_consensus.cc:348] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.541121 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.541950 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.542491 20446 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 0.
I20240506 15:13:33.539839 20563 raft_consensus.cc:374] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.543350 20563 raft_consensus.cc:729] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.544214 20563 consensus_queue.cc:260] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.545796 20561 tablet_bootstrap.cc:654] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.548858 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.549719 20563 ts_tablet_manager.cc:1398] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20240506 15:13:33.549881 20520 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 0.
I20240506 15:13:33.551528 20563 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.552939 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.553903 20518 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 0.
I20240506 15:13:33.556632 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.551645 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.560564 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.561846 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
W20240506 15:13:33.563061 20337 leader_election.cc:341] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.563807 20563 tablet_bootstrap.cc:654] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.563802 20337 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.565022 20566 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.565366 20566 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.565724 20566 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:13:33.570674 20318 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:13:33.573899 20338 leader_election.cc:302] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:33.577328 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.578032 20447 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 0.
W20240506 15:13:33.578872 20337 leader_election.cc:341] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:13:33.579600 20337 leader_election.cc:341] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.580899 20486 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.582049 20568 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.582424 20568 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.582906 20568 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.592430 20584 raft_consensus.cc:2781] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.615247 20584 raft_consensus.cc:482] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.615697 20584 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.593324 20561 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.616705 20562 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.617070 20561 ts_tablet_manager.cc:1367] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.082s user 0.027s sys 0.005s
I20240506 15:13:33.617175 20562 ts_tablet_manager.cc:1367] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.097s user 0.033s sys 0.052s
W20240506 15:13:33.594211 20338 leader_election.cc:341] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.619777 20561 raft_consensus.cc:348] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.610548 20568 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.620608 20561 raft_consensus.cc:374] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.620471 20338 leader_election.cc:302] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f; no voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.620990 20561 raft_consensus.cc:729] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.599551 20566 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.616705 20563 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.621971 20568 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.622161 20563 ts_tablet_manager.cc:1367] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.071s user 0.022s sys 0.039s
I20240506 15:13:33.619920 20562 raft_consensus.cc:348] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.622515 20562 raft_consensus.cc:374] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.622838 20562 raft_consensus.cc:729] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.622889 20566 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.623492 20566 raft_consensus.cc:2726] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:33.623606 20562 consensus_queue.cc:260] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.624377 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:33.624945 20563 raft_consensus.cc:348] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.625715 20563 raft_consensus.cc:374] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.626060 20563 raft_consensus.cc:729] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.623507 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:33.627159 20563 consensus_queue.cc:260] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.628872 20562 ts_tablet_manager.cc:1398] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20240506 15:13:33.629900 20562 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.629818 20584 raft_consensus.cc:504] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.630028 20561 consensus_queue.cc:260] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.625226 20446 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.632769 20561 ts_tablet_manager.cc:1398] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240506 15:13:33.633709 20561 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.636442 20562 tablet_bootstrap.cc:654] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.639954 20561 tablet_bootstrap.cc:654] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.629900 20563 ts_tablet_manager.cc:1398] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240506 15:13:33.641685 20563 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.641829 20570 raft_consensus.cc:482] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.642416 20570 raft_consensus.cc:504] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.646750 20570 leader_election.cc:288] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.653558 20563 tablet_bootstrap.cc:654] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.656185 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
W20240506 15:13:33.662068 20337 leader_election.cc:341] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.658938 20584 leader_election.cc:288] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Requested vote from peers 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.678227 20563 tablet_bootstrap.cc:492] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.678968 20563 ts_tablet_manager.cc:1367] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.011s
I20240506 15:13:33.681697 20446 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 1.
I20240506 15:13:33.682752 20563 raft_consensus.cc:348] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.684926 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.690639 20372 raft_consensus.cc:2379] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a0a151b64f54c71a48043fe8c832de8 in current term 1: Already voted for candidate 1a1b2df2b681456b9e3e14ab555ac89f in this term.
I20240506 15:13:33.690650 20337 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.686070 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:33.692111 20584 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:33.692538 20563 raft_consensus.cc:374] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.693042 20563 raft_consensus.cc:729] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.693569 20584 raft_consensus.cc:686] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:33.694065 20563 consensus_queue.cc:260] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.694631 20584 consensus_queue.cc:237] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.696970 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.697651 20372 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.697841 20563 ts_tablet_manager.cc:1398] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.018s user 0.008s sys 0.004s
I20240506 15:13:33.700114 20563 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.700042 20411 leader_election.cc:302] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.701385 20570 raft_consensus.cc:2781] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.701897 20570 raft_consensus.cc:482] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.702386 20570 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.705639 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.706322 20519 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.710656 20570 raft_consensus.cc:504] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.714278 20563 tablet_bootstrap.cc:654] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.717924 20485 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8; no voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.723985 20568 raft_consensus.cc:2726] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:13:33.725529 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:33.726123 20519 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.726274 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:33.726881 20371 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.733733 20570 leader_election.cc:288] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
W20240506 15:13:33.721995 20338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42587, user_credentials={real_user=slave}} blocked reactor thread for 127880us
I20240506 15:13:33.733644 20519 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
W20240506 15:13:33.734757 20338 leader_election.cc:341] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.735170 20338 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f; no voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.735628 20413 leader_election.cc:302] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.736735 20585 raft_consensus.cc:2726] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:33.738523 20563 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.719550 20267 catalog_manager.cc:5484] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 1 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:33.737291 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:33.739334 20563 ts_tablet_manager.cc:1367] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.002s
I20240506 15:13:33.739780 20521 raft_consensus.cc:2379] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1a1b2df2b681456b9e3e14ab555ac89f in current term 1: Already voted for candidate 7a0a151b64f54c71a48043fe8c832de8 in this term.
I20240506 15:13:33.733644 20371 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:33.738387 20570 raft_consensus.cc:2781] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:33.738740 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:33.741407 20519 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.747073 20561 tablet_bootstrap.cc:492] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.747749 20561 ts_tablet_manager.cc:1367] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.114s user 0.018s sys 0.028s
I20240506 15:13:33.751868 20519 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 1.
I20240506 15:13:33.743911 20563 raft_consensus.cc:348] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.753482 20563 raft_consensus.cc:374] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.753568 20338 leader_election.cc:302] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters: c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.754004 20563 raft_consensus.cc:729] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.755504 20585 raft_consensus.cc:2781] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:33.754988 20563 consensus_queue.cc:260] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.756189 20585 raft_consensus.cc:686] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:33.756675 20570 raft_consensus.cc:686] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:33.752156 20561 raft_consensus.cc:348] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.757521 20561 raft_consensus.cc:374] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.757467 20585 consensus_queue.cc:237] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.758031 20561 raft_consensus.cc:729] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.757901 20570 consensus_queue.cc:237] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.759114 20561 consensus_queue.cc:260] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.761657 20561 ts_tablet_manager.cc:1398] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.014s user 0.005s sys 0.005s
I20240506 15:13:33.762584 20561 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.765769 20562 tablet_bootstrap.cc:492] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.767778 20562 ts_tablet_manager.cc:1367] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.138s user 0.031s sys 0.078s
I20240506 15:13:33.773181 20562 raft_consensus.cc:348] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.773941 20562 raft_consensus.cc:374] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.775590 20595 raft_consensus.cc:482] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.776051 20584 raft_consensus.cc:482] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.776151 20595 raft_consensus.cc:504] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.774463 20562 raft_consensus.cc:729] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.779255 20563 ts_tablet_manager.cc:1398] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.039s user 0.004s sys 0.007s
I20240506 15:13:33.778841 20562 consensus_queue.cc:260] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.780212 20563 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.780673 20561 tablet_bootstrap.cc:654] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.781196 20562 ts_tablet_manager.cc:1398] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20240506 15:13:33.776835 20584 raft_consensus.cc:504] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.782162 20562 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.783331 20584 leader_election.cc:288] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.786556 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.787428 20268 catalog_manager.cc:5484] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 1 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:33.787775 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
W20240506 15:13:33.789561 20337 leader_election.cc:341] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.792866 20562 tablet_bootstrap.cc:654] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
W20240506 15:13:33.794023 20338 leader_election.cc:341] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.794502 20338 leader_election.cc:302] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f; no voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.796124 20584 raft_consensus.cc:2726] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:33.799266 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.801131 20563 tablet_bootstrap.cc:654] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.801436 20371 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.803201 20411 leader_election.cc:302] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.805611 20570 raft_consensus.cc:2781] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.806123 20570 raft_consensus.cc:482] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.806566 20570 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.803202 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.809299 20519 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.796838 20595 leader_election.cc:288] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.821034 20568 raft_consensus.cc:482] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.821702 20568 raft_consensus.cc:504] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.826658 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
W20240506 15:13:33.828660 20486 leader_election.cc:341] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.828784 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.829548 20568 leader_election.cc:288] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.820055 20570 raft_consensus.cc:504] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
W20240506 15:13:33.830861 20485 leader_election.cc:341] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:13:33.831470 20485 leader_election.cc:302] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8; no voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:33.833557 20570 leader_election.cc:288] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.833634 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:33.833912 20568 raft_consensus.cc:2726] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:13:33.834306 20371 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.821220 20562 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.835518 20562 ts_tablet_manager.cc:1367] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.015s
I20240506 15:13:33.836776 20561 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.836652 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:33.837260 20561 ts_tablet_manager.cc:1367] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.005s
I20240506 15:13:33.838474 20562 raft_consensus.cc:348] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.839259 20562 raft_consensus.cc:374] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.839638 20562 raft_consensus.cc:729] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.837275 20519 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.839934 20561 raft_consensus.cc:348] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.840752 20561 raft_consensus.cc:374] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.840476 20562 consensus_queue.cc:260] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.841027 20561 raft_consensus.cc:729] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.842027 20561 consensus_queue.cc:260] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.843123 20562 ts_tablet_manager.cc:1398] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240506 15:13:33.843971 20561 ts_tablet_manager.cc:1398] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240506 15:13:33.844153 20562 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.844873 20561 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.845319 20268 catalog_manager.cc:5484] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 1 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:33.848721 20561 tablet_bootstrap.cc:654] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.850984 20562 tablet_bootstrap.cc:654] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.861891 20371 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:33.863626 20562 tablet_bootstrap.cc:492] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.862779 20561 tablet_bootstrap.cc:492] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.863392 20519 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:33.864704 20561 ts_tablet_manager.cc:1367] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20240506 15:13:33.865271 20413 leader_election.cc:302] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.865834 20563 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.864190 20562 ts_tablet_manager.cc:1367] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.016s
I20240506 15:13:33.866289 20570 raft_consensus.cc:2781] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:33.866793 20570 raft_consensus.cc:686] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:33.867518 20561 raft_consensus.cc:348] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.867561 20570 consensus_queue.cc:237] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.868340 20561 raft_consensus.cc:374] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.868634 20561 raft_consensus.cc:729] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.868846 20562 raft_consensus.cc:348] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.869508 20562 raft_consensus.cc:374] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.869246 20561 consensus_queue.cc:260] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.869747 20562 raft_consensus.cc:729] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.870451 20562 consensus_queue.cc:260] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:33.871313 20561 ts_tablet_manager.cc:1398] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 15:13:33.872232 20561 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: Bootstrap starting.
I20240506 15:13:33.872879 20562 ts_tablet_manager.cc:1398] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:13:33.873687 20562 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.866271 20563 ts_tablet_manager.cc:1367] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.011s
I20240506 15:13:33.877312 20561 tablet_bootstrap.cc:654] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.877727 20563 raft_consensus.cc:348] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.879047 20563 raft_consensus.cc:374] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.879345 20563 raft_consensus.cc:729] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.879608 20562 tablet_bootstrap.cc:654] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.880201 20563 consensus_queue.cc:260] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.890964 20570 raft_consensus.cc:482] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.891698 20570 raft_consensus.cc:504] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.896229 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:33.896234 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.897111 20371 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.897099 20519 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:33.897076 20267 catalog_manager.cc:5484] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 1 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:33.898607 20411 leader_election.cc:302] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.900872 20570 leader_election.cc:288] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.901271 20595 raft_consensus.cc:2781] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.901803 20595 raft_consensus.cc:482] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.902225 20595 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.913393 20595 raft_consensus.cc:504] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.922142 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:33.923027 20371 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.916800 20563 ts_tablet_manager.cc:1398] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20240506 15:13:33.928349 20563 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.931552 20371 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:33.933557 20411 leader_election.cc:302] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:33.933616 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:33.932509 20595 leader_election.cc:288] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:33.934286 20519 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.934499 20570 raft_consensus.cc:2781] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:33.935230 20570 raft_consensus.cc:686] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:33.936355 20570 consensus_queue.cc:237] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.938903 20563 tablet_bootstrap.cc:654] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.941076 20562 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.941509 20519 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:33.941681 20562 ts_tablet_manager.cc:1367] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.068s user 0.032s sys 0.020s
I20240506 15:13:33.941097 20561 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: No bootstrap required, opened a new log
I20240506 15:13:33.944468 20562 raft_consensus.cc:348] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.945276 20561 ts_tablet_manager.cc:1367] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent bootstrapping tablet: real 0.073s user 0.030s sys 0.036s
I20240506 15:13:33.945441 20562 raft_consensus.cc:374] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.945770 20562 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.946357 20562 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.948055 20562 ts_tablet_manager.cc:1398] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 15:13:33.948813 20562 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.948521 20561 raft_consensus.cc:348] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.949398 20561 raft_consensus.cc:374] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.949716 20561 raft_consensus.cc:729] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.950456 20561 consensus_queue.cc:260] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.952555 20561 ts_tablet_manager.cc:1398] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240506 15:13:33.954741 20562 tablet_bootstrap.cc:654] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.955112 20563 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.956213 20268 catalog_manager.cc:5484] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 1 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:33.957324 20563 ts_tablet_manager.cc:1367] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.003s
I20240506 15:13:33.960243 20563 raft_consensus.cc:348] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.960985 20563 raft_consensus.cc:374] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.961241 20563 raft_consensus.cc:729] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.961437 20562 tablet_bootstrap.cc:492] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.961921 20562 ts_tablet_manager.cc:1367] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240506 15:13:33.962036 20563 consensus_queue.cc:260] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.964699 20562 raft_consensus.cc:348] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.965404 20562 raft_consensus.cc:374] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.965576 20563 ts_tablet_manager.cc:1398] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240506 15:13:33.965716 20562 raft_consensus.cc:729] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.966634 20563 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.966380 20562 consensus_queue.cc:260] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.968601 20562 ts_tablet_manager.cc:1398] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:13:33.969643 20562 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:33.970573 20568 raft_consensus.cc:482] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:33.971062 20568 raft_consensus.cc:504] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.972858 20563 tablet_bootstrap.cc:654] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.974607 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:33.975143 20371 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 0.
I20240506 15:13:33.975708 20562 tablet_bootstrap.cc:654] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.976207 20485 leader_election.cc:302] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:33.976779 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:33.978005 20446 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 0.
I20240506 15:13:33.977231 20568 leader_election.cc:288] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.981298 20568 raft_consensus.cc:2781] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:33.981704 20568 raft_consensus.cc:482] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:33.982115 20568 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.982964 20563 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:33.983414 20563 ts_tablet_manager.cc:1367] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20240506 15:13:33.986110 20563 raft_consensus.cc:348] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.986783 20563 raft_consensus.cc:374] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:33.987059 20563 raft_consensus.cc:729] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:33.987702 20563 consensus_queue.cc:260] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.989794 20563 ts_tablet_manager.cc:1398] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240506 15:13:33.989677 20568 raft_consensus.cc:504] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:33.990677 20563 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: Bootstrap starting.
I20240506 15:13:33.991652 20568 leader_election.cc:288] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:33.993712 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:33.994519 20446 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.995558 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:33.996150 20371 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:33.996485 20563 tablet_bootstrap.cc:654] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:33.998311 20562 tablet_bootstrap.cc:492] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:33.998781 20562 ts_tablet_manager.cc:1367] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.001s
I20240506 15:13:34.001648 20446 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 1.
I20240506 15:13:34.002076 20562 raft_consensus.cc:348] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.003093 20486 leader_election.cc:302] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.003661 20562 raft_consensus.cc:374] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:34.004038 20562 raft_consensus.cc:729] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:34.004205 20613 raft_consensus.cc:2781] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:34.004806 20613 raft_consensus.cc:686] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:34.005180 20562 consensus_queue.cc:260] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.005678 20613 consensus_queue.cc:237] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.008231 20562 ts_tablet_manager.cc:1398] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240506 15:13:34.009516 20562 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: Bootstrap starting.
I20240506 15:13:34.006131 20371 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 1.
I20240506 15:13:34.016960 20562 tablet_bootstrap.cc:654] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: Neither blocks nor log segments found. Creating new log.
I20240506 15:13:34.032194 20563 tablet_bootstrap.cc:492] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: No bootstrap required, opened a new log
I20240506 15:13:34.032677 20563 ts_tablet_manager.cc:1367] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.019s
I20240506 15:13:34.035118 20562 tablet_bootstrap.cc:492] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: No bootstrap required, opened a new log
I20240506 15:13:34.035658 20562 ts_tablet_manager.cc:1367] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.002s
I20240506 15:13:34.035328 20563 raft_consensus.cc:348] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.036039 20563 raft_consensus.cc:374] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:34.036365 20563 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Initialized, Role: FOLLOWER
I20240506 15:13:34.037006 20563 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.037359 20266 catalog_manager.cc:5484] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:34.039124 20563 ts_tablet_manager.cc:1398] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240506 15:13:34.038883 20562 raft_consensus.cc:348] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.039564 20562 raft_consensus.cc:374] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:13:34.039815 20562 raft_consensus.cc:729] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Initialized, Role: FOLLOWER
I20240506 15:13:34.040480 20562 consensus_queue.cc:260] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.041843 20562 ts_tablet_manager.cc:1398] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:13:34.053838 20613 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:34.054273 20613 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.055868 20613 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:34.056581 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:34.057190 20371 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 0.
I20240506 15:13:34.057159 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:34.057976 20446 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 0.
I20240506 15:13:34.058319 20485 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:34.059051 20613 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:34.059394 20613 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:34.059705 20613 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.064832 20613 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.067539 20613 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:34.067912 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:34.068610 20371 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.070046 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.070658 20446 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.072327 20570 raft_consensus.cc:482] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:34.072712 20570 raft_consensus.cc:504] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.074282 20570 leader_election.cc:288] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.075048 20371 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 1.
I20240506 15:13:34.075139 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:34.075814 20372 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:34.076136 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:34.076200 20485 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:34.076866 20519 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:34.076978 20411 leader_election.cc:302] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.077272 20446 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 1.
I20240506 15:13:34.077760 20570 raft_consensus.cc:2781] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:34.078015 20613 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:34.078142 20570 raft_consensus.cc:482] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:34.078591 20570 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.078593 20613 raft_consensus.cc:686] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:34.079560 20613 consensus_queue.cc:237] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.083879 20570 raft_consensus.cc:504] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.086114 20570 leader_election.cc:288] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.086441 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:34.087028 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.087105 20371 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.087472 20519 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.088614 20570 raft_consensus.cc:482] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:34.088161 20268 catalog_manager.cc:5484] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:34.089061 20570 raft_consensus.cc:504] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.092065 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" is_pre_election: true
I20240506 15:13:34.092669 20570 leader_election.cc:288] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.092142 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:34.092885 20372 raft_consensus.cc:2454] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:34.093376 20521 raft_consensus.cc:2454] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 0.
I20240506 15:13:34.094105 20411 leader_election.cc:302] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.094321 20371 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:34.095829 20411 leader_election.cc:302] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.097101 20595 raft_consensus.cc:2781] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:34.097703 20595 raft_consensus.cc:482] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:34.098093 20595 raft_consensus.cc:3037] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.098137 20519 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:34.098277 20570 raft_consensus.cc:2781] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:34.100067 20584 raft_consensus.cc:482] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:13:34.100096 20570 raft_consensus.cc:686] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:34.100749 20584 raft_consensus.cc:504] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.101096 20570 consensus_queue.cc:237] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.102772 20584 leader_election.cc:288] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.103883 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8" is_pre_election: true
I20240506 15:13:34.104604 20519 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 0.
I20240506 15:13:34.104694 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" is_pre_election: true
I20240506 15:13:34.105207 20595 raft_consensus.cc:504] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.105566 20446 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 0.
I20240506 15:13:34.106985 20337 leader_election.cc:302] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.107614 20595 leader_election.cc:288] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.108507 20584 raft_consensus.cc:2781] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:13:34.108907 20584 raft_consensus.cc:482] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:13:34.109397 20584 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.110749 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:34.111348 20371 raft_consensus.cc:3037] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.113648 20268 catalog_manager.cc:5484] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 1 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.115425 20584 raft_consensus.cc:504] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.108888 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f94eb5f7a4502a4ac643e466d4cf9" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.117971 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.118240 20519 raft_consensus.cc:3037] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.118533 20446 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.118485 20371 raft_consensus.cc:2454] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:34.119956 20411 leader_election.cc:302] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.120909 20570 raft_consensus.cc:2781] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:34.121405 20570 raft_consensus.cc:686] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:34.122306 20570 consensus_queue.cc:237] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.124619 20584 leader_election.cc:288] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.125880 20446 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 1.
I20240506 15:13:34.126360 20519 raft_consensus.cc:2454] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 1.
I20240506 15:13:34.126981 20337 leader_election.cc:302] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.128161 20584 raft_consensus.cc:2781] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:13:34.129179 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.129666 20584 raft_consensus.cc:686] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.129838 20518 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.130760 20585 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.130548 20584 consensus_queue.cc:237] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.135989 20268 catalog_manager.cc:5484] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 1 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.137528 20518 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 1.
I20240506 15:13:34.152417 20267 catalog_manager.cc:5484] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 1 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.155207 20585 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.174615 20585 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.182547 20446 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:13:34.191076 20585 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.211290 20570 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.221863 20570 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:34.347975 20521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:58320
I20240506 15:13:34.348613 20521 raft_consensus.cc:595] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:34.353498 20521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:58320
I20240506 15:13:34.353979 20521 raft_consensus.cc:595] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:34.357136 20631 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.362043 20631 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.370568 20446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:55098
I20240506 15:13:34.371165 20446 raft_consensus.cc:595] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:34.377099 20446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:55098
I20240506 15:13:34.377605 20446 raft_consensus.cc:595] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:34.379731 20631 raft_consensus.cc:979] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.380208 20595 raft_consensus.cc:1067] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.383319 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52846
I20240506 15:13:34.385370 20371 raft_consensus.cc:482] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.385700 20371 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.386286 20595 consensus_queue.cc:1035] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.397413 20371 raft_consensus.cc:504] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.399814 20370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:52858
I20240506 15:13:34.400497 20370 raft_consensus.cc:595] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Received request to transfer leadership to TS c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:34.409188 20595 consensus_queue.cc:1035] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.411727 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.412398 20446 raft_consensus.cc:3032] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.412720 20446 raft_consensus.cc:729] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:34.413476 20446 consensus_queue.cc:260] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.414664 20446 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.417816 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.418634 20518 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.421651 20446 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.422482 20372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:52858
I20240506 15:13:34.422648 20337 leader_election.cc:302] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.423171 20372 raft_consensus.cc:595] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Received request to transfer leadership to TS 7a0a151b64f54c71a48043fe8c832de8
I20240506 15:13:34.423409 20585 raft_consensus.cc:2781] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.423895 20585 raft_consensus.cc:686] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.424751 20585 consensus_queue.cc:237] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.426090 20518 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.427354 20371 leader_election.cc:288] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.434477 20270 catalog_manager.cc:5484] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.449744 20613 consensus_queue.cc:1035] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.460093 20568 consensus_queue.cc:1035] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.467206 20613 raft_consensus.cc:979] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.467622 20568 raft_consensus.cc:1067] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.469218 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52832
I20240506 15:13:34.469755 20371 raft_consensus.cc:482] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.470046 20371 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.474781 20371 raft_consensus.cc:504] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.476315 20371 leader_election.cc:288] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.477353 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.477900 20446 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.477644 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4dc049405b4c75a4e8cf154ae8f7fa" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.478679 20521 raft_consensus.cc:3032] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.479019 20521 raft_consensus.cc:729] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:34.479804 20521 consensus_queue.cc:260] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.481137 20521 raft_consensus.cc:3037] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.482901 20446 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.483851 20337 leader_election.cc:302] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.484695 20585 raft_consensus.cc:2781] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.485056 20585 raft_consensus.cc:686] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.485806 20585 consensus_queue.cc:237] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.486936 20521 raft_consensus.cc:2454] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.492120 20270 catalog_manager.cc:5484] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.507704 20595 consensus_queue.cc:1035] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.509652 20568 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.519217 20593 consensus_queue.cc:1035] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.532291 20613 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.541652 20642 raft_consensus.cc:979] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.542114 20613 raft_consensus.cc:1067] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.543835 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52832
I20240506 15:13:34.544461 20371 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.544878 20371 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.549553 20371 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.552057 20371 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.552750 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.553537 20446 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.554920 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.555546 20518 raft_consensus.cc:3032] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.555980 20518 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:34.556762 20518 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.558255 20518 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.560467 20446 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.561725 20337 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.562408 20585 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.562897 20585 raft_consensus.cc:686] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.563728 20585 consensus_queue.cc:237] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.564620 20518 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.571489 20270 catalog_manager.cc:5484] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.597743 20585 raft_consensus.cc:979] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:34.598163 20566 raft_consensus.cc:1067] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Signalling peer c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:34.599472 20446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:55092
I20240506 15:13:34.600064 20446 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.600512 20446 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.605157 20446 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.606808 20446 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 2 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.607614 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:34.607838 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.608130 20371 raft_consensus.cc:3032] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.608522 20371 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.608450 20518 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.609347 20371 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.610687 20371 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.611718 20593 consensus_queue.cc:1035] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.614820 20518 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 2.
I20240506 15:13:34.617009 20371 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 2.
I20240506 15:13:34.616125 20413 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.618708 20593 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.619184 20593 raft_consensus.cc:686] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:34.620012 20593 consensus_queue.cc:237] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.624353 20570 consensus_queue.cc:1035] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:13:34.627954 20270 catalog_manager.cc:5484] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 1 to 2, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 2 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.683741 20566 consensus_queue.cc:1035] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.693301 20585 consensus_queue.cc:1035] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:13:34.695775 20570 raft_consensus.cc:979] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.696224 20570 raft_consensus.cc:1067] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:34.698249 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52846
I20240506 15:13:34.698920 20371 raft_consensus.cc:482] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.699347 20371 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.701773 20628 raft_consensus.cc:979] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:34.702250 20566 raft_consensus.cc:1067] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Signalling peer 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:34.703927 20521 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:58296
I20240506 15:13:34.704447 20521 raft_consensus.cc:482] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:34.704742 20521 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.707185 20371 raft_consensus.cc:504] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.708897 20371 leader_election.cc:288] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:34.709854 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:34.709791 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.710557 20518 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.710614 20446 raft_consensus.cc:3032] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.710695 20521 raft_consensus.cc:504] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.711129 20446 raft_consensus.cc:729] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:34.711946 20446 consensus_queue.cc:260] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.713371 20446 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.714390 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:34.714854 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e41659303f74c5084ff0a5d2ae0ac04" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:34.715078 20371 raft_consensus.cc:3032] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:34.715404 20371 raft_consensus.cc:729] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.715409 20447 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.716365 20371 consensus_queue.cc:260] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.717515 20371 raft_consensus.cc:3037] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:34.717620 20518 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.713757 20521 leader_election.cc:288] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 2 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:34.721531 20446 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:34.722374 20447 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 2.
I20240506 15:13:34.722901 20337 leader_election.cc:302] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.723737 20566 raft_consensus.cc:2781] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.724050 20486 leader_election.cc:302] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:34.724345 20566 raft_consensus.cc:686] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:34.725219 20613 raft_consensus.cc:2781] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:34.725382 20371 raft_consensus.cc:2454] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 2.
I20240506 15:13:34.725199 20566 consensus_queue.cc:237] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.725884 20613 raft_consensus.cc:686] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:34.726725 20613 consensus_queue.cc:237] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:34.732997 20267 catalog_manager.cc:5484] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:34.734004 20270 catalog_manager.cc:5484] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 1 to 2, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 2 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:34.871274 20447 raft_consensus.cc:1261] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:34.872481 20566 consensus_queue.cc:1035] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:34.879310 20521 raft_consensus.cc:1261] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:34.880774 20585 consensus_queue.cc:1035] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:34.928151 20447 raft_consensus.cc:1261] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:34.929422 20585 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:34.937770 20521 raft_consensus.cc:1261] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:34.939502 20585 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.026402 20447 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.027865 20585 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.034402 20521 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.035939 20652 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.097010 20521 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.098152 20595 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.105890 20371 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.107439 20570 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.112308 20447 raft_consensus.cc:1261] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.113658 20566 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.121974 20521 raft_consensus.cc:1261] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.123417 20652 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.170629 20371 raft_consensus.cc:1261] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.171731 20613 consensus_queue.cc:1035] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:35.178634 20447 raft_consensus.cc:1261] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:35.180197 20613 consensus_queue.cc:1035] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:37.454025 19726 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240506 15:13:37.454910 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:37.477138 20371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:37.477978 20371 raft_consensus.cc:595] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Received request to transfer leadership to TS 7a0a151b64f54c71a48043fe8c832de8
I20240506 15:13:37.481338 20371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:37.481873 20371 raft_consensus.cc:595] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Received request to transfer leadership to TS 7a0a151b64f54c71a48043fe8c832de8
I20240506 15:13:37.482786 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240506 15:13:37.483234 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:37.664120 20657 raft_consensus.cc:979] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:37.664686 20661 raft_consensus.cc:1067] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Signalling peer 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:37.666188 20518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:58296
I20240506 15:13:37.666772 20518 raft_consensus.cc:482] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:37.667129 20518 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:37.672227 20518 raft_consensus.cc:504] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:37.674525 20518 leader_election.cc:288] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 2 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:37.675472 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:37.675446 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:37.676183 20447 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:37.676272 20371 raft_consensus.cc:3032] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:37.676741 20371 raft_consensus.cc:729] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:37.677603 20371 consensus_queue.cc:260] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:37.678982 20371 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:37.682016 20447 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 2.
I20240506 15:13:37.683125 20486 leader_election.cc:302] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:37.683313 20371 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 2.
I20240506 15:13:37.684057 20656 raft_consensus.cc:2781] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:37.685254 20656 raft_consensus.cc:686] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:37.685926 20656 consensus_queue.cc:237] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:37.692643 20266 catalog_manager.cc:5484] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 1 to 2, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 2 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:37.842327 20657 raft_consensus.cc:979] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:37.842876 20661 raft_consensus.cc:1067] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Signalling peer 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:37.844623 20518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:58296
I20240506 15:13:37.845139 20518 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:37.845424 20518 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:37.849956 20518 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:37.851565 20518 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 3 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:37.852480 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:37.852780 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:37.853091 20371 raft_consensus.cc:3032] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:37.853332 20447 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:37.853461 20371 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:37.854189 20371 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:37.855340 20371 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:37.860188 20447 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 3.
I20240506 15:13:37.861465 20371 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 3.
I20240506 15:13:37.861699 20486 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:37.862767 20656 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:37.863461 20656 raft_consensus.cc:686] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:37.864362 20656 consensus_queue.cc:237] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:37.872105 20266 catalog_manager.cc:5484] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 2 to 3, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:38.106801 20447 raft_consensus.cc:1261] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:38.107976 20656 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:38.123152 20371 raft_consensus.cc:1261] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:38.125034 20656 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:13:38.262651 20371 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:38.263754 20656 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:38.273059 20447 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:38.274720 20676 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:38.484179 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:38.490410 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:38.490819 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:39.491685 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:39.497069 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:39.497503 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:40.498450 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:40.504745 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:40.505290 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:41.515983 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:41.526738 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:41.527179 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:42.527993 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:42.533293 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:42.533694 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:43.534510 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:43.540230 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:43.540665 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:44.541571 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:44.546938 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:44.547438 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:45.548295 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:45.554730 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:45.555090 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:46.556079 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:46.562427 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:46.562875 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:47.593477 19726 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240506 15:13:47.598008 20521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:58320
I20240506 15:13:47.598943 20521 raft_consensus.cc:595] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:47.604876 20521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:58320
I20240506 15:13:47.605424 20521 raft_consensus.cc:595] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:47.619900 20447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:55098
I20240506 15:13:47.620496 20447 raft_consensus.cc:595] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:47.625785 20447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:55098
I20240506 15:13:47.626408 20447 raft_consensus.cc:595] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:47.630987 20447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
mode: GRACEFUL
new_leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:55098
I20240506 15:13:47.631433 20447 raft_consensus.cc:595] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Received request to transfer leadership to TS 1a1b2df2b681456b9e3e14ab555ac89f
I20240506 15:13:47.692565 20683 raft_consensus.cc:979] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.693166 20681 raft_consensus.cc:1067] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.694782 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52832
I20240506 15:13:47.695386 20371 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:47.695906 20371 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:47.698082 20681 raft_consensus.cc:979] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.698652 20570 raft_consensus.cc:979] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.698688 20683 raft_consensus.cc:1067] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.699615 20684 raft_consensus.cc:1067] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.700853 20372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52832
I20240506 15:13:47.701414 20372 raft_consensus.cc:482] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:47.701472 20370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52846
I20240506 15:13:47.701789 20372 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.701897 20370 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:47.702342 20370 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.704385 20371 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.708050 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:47.709113 20521 raft_consensus.cc:3032] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:13:47.708014 20370 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.709470 20521 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:47.710109 20372 raft_consensus.cc:504] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:47.711428 20370 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 3 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:47.708048 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:47.712146 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:47.712461 20372 leader_election.cc:288] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 3 election: Requested vote from peers 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:47.710690 20521 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.712882 20446 raft_consensus.cc:3032] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:47.713267 20446 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:47.713354 20521 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:47.713575 20371 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 4 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:47.714071 20446 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.715329 20446 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.717056 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:47.717630 20518 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.720816 20521 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 4.
I20240506 15:13:47.722352 20338 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:47.723486 20661 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:13:47.712157 20447 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:47.725026 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:47.726027 20519 raft_consensus.cc:3032] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:47.726476 20519 raft_consensus.cc:729] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:47.727713 20519 consensus_queue.cc:260] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:47.729895 20519 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.732339 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6e40f376fc4c8f8ff453ea1830a594" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:47.733017 20445 raft_consensus.cc:3037] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:47.723805 20518 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 3.
I20240506 15:13:47.737882 20446 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 3.
I20240506 15:13:47.737697 20338 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:47.740677 20661 raft_consensus.cc:686] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:47.740815 20445 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 3.
I20240506 15:13:47.741250 20685 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:47.741788 20661 consensus_queue.cc:237] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.742321 20447 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 4.
I20240506 15:13:47.743533 20519 raft_consensus.cc:2454] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 3.
I20240506 15:13:47.743469 20337 leader_election.cc:302] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:47.743466 20685 raft_consensus.cc:686] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:47.745275 20685 consensus_queue.cc:237] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.746011 20687 raft_consensus.cc:2781] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:47.757079 20687 raft_consensus.cc:686] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:47.757726 20266 catalog_manager.cc:5484] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 3 to 4, leader changed from 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 4 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:47.758080 20687 consensus_queue.cc:237] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } }
I20240506 15:13:47.775504 20266 catalog_manager.cc:5484] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 2 to 3, leader changed from c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 3 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:47.776690 20266 catalog_manager.cc:5484] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 2 to 3, leader changed from 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 3 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:47.842257 20570 raft_consensus.cc:979] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.843642 20684 raft_consensus.cc:1067] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.844431 20696 raft_consensus.cc:979] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: : Instructing follower 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.844913 20684 raft_consensus.cc:1067] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Signalling peer 1a1b2df2b681456b9e3e14ab555ac89f to start an election
I20240506 15:13:47.845336 20370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52846
I20240506 15:13:47.845890 20370 raft_consensus.cc:482] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:47.846181 20370 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.846381 20371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
from {username='slave'} at 127.0.0.1:52846
I20240506 15:13:47.846889 20371 raft_consensus.cc:482] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:47.847240 20371 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.852895 20370 raft_consensus.cc:504] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.853400 20371 raft_consensus.cc:504] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.854923 20371 leader_election.cc:288] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:47.854926 20370 leader_election.cc:288] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Requested vote from peers c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:47.855713 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:47.856106 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:47.856456 20447 raft_consensus.cc:3032] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:47.856613 20445 raft_consensus.cc:3032] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:13:47.856557 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c21597f811461ba94aac038acf0a0b" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:47.856983 20445 raft_consensus.cc:729] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:47.856850 20447 raft_consensus.cc:729] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:47.857331 20518 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.857887 20447 consensus_queue.cc:260] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.857863 20445 consensus_queue.cc:260] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.859163 20447 raft_consensus.cc:3037] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.859362 20445 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.859228 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3f567b0fef40059673b90d4899f471" candidate_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:47.860550 20519 raft_consensus.cc:3037] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:13:47.864271 20518 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:47.865794 20338 leader_election.cc:302] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, 7a0a151b64f54c71a48043fe8c832de8; no voters:
I20240506 15:13:47.865957 20445 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:47.866658 20687 raft_consensus.cc:2781] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:47.867026 20337 leader_election.cc:302] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1b2df2b681456b9e3e14ab555ac89f, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:47.867223 20519 raft_consensus.cc:2454] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:47.867842 20687 raft_consensus.cc:686] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:47.867990 20447 raft_consensus.cc:2454] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1b2df2b681456b9e3e14ab555ac89f in term 2.
I20240506 15:13:47.868357 20689 raft_consensus.cc:2781] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:13:47.868899 20689 raft_consensus.cc:686] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Leader. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:47.868932 20687 consensus_queue.cc:237] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.869969 20689 consensus_queue.cc:237] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:47.876991 20266 catalog_manager.cc:5484] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:47.887591 20266 catalog_manager.cc:5484] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f reported cstate change: term changed from 1 to 2, leader changed from c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130) to 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129). New cstate: current_term: 2 leader_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:48.135044 20519 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:48.137094 20689 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:48.151062 20447 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:48.152801 20689 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:48.207206 20519 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:48.208515 20661 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:48.218271 20447 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:48.222664 20661 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:48.242379 20447 raft_consensus.cc:1261] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:48.243636 20661 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:48.253319 20519 raft_consensus.cc:1261] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:48.256975 20689 consensus_queue.cc:1035] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:48.300371 20445 raft_consensus.cc:1261] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:48.302136 20661 consensus_queue.cc:1035] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:48.303452 20519 raft_consensus.cc:1261] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:48.304905 20661 consensus_queue.cc:1035] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:13:48.359599 20519 raft_consensus.cc:1261] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:48.360994 20685 consensus_queue.cc:1035] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:48.369606 20445 raft_consensus.cc:1261] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Refusing update from remote peer 1a1b2df2b681456b9e3e14ab555ac89f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:13:48.371366 20661 consensus_queue.cc:1035] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:13:50.670195 19726 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240506 15:13:50.670881 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:50.679167 20370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:50.679778 20370 raft_consensus.cc:595] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Received request to transfer leadership to TS 7a0a151b64f54c71a48043fe8c832de8
I20240506 15:13:50.682825 20370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:50.683333 20370 raft_consensus.cc:595] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Received request to transfer leadership to TS c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:50.685669 20370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:50.686144 20370 raft_consensus.cc:595] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 LEADER]: Received request to transfer leadership to TS 7a0a151b64f54c71a48043fe8c832de8
I20240506 15:13:50.688637 20370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2"
dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
mode: GRACEFUL
new_leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:52868
I20240506 15:13:50.689136 20370 raft_consensus.cc:595] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Received request to transfer leadership to TS c8f8d10dd45c4db99cb13d82bd75b0ab
I20240506 15:13:50.690191 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240506 15:13:50.690704 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:50.713405 20689 raft_consensus.cc:979] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:50.714005 20689 raft_consensus.cc:1067] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Signalling peer c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:50.715795 20445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:55092
I20240506 15:13:50.716380 20445 raft_consensus.cc:482] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:50.716698 20445 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.721478 20445 raft_consensus.cc:504] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.723529 20445 leader_election.cc:288] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 3 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:50.724825 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:50.724825 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda181b266b74a7388b5a86923942cb2" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:50.725621 20518 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.725618 20370 raft_consensus.cc:3032] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:50.726308 20370 raft_consensus.cc:729] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:50.727301 20370 consensus_queue.cc:260] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.728556 20370 raft_consensus.cc:3037] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.730799 20518 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 3.
I20240506 15:13:50.732211 20413 leader_election.cc:302] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:50.732971 20684 raft_consensus.cc:2781] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:50.734709 20684 raft_consensus.cc:686] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:50.736655 20370 raft_consensus.cc:2454] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 3.
I20240506 15:13:50.737021 20684 consensus_queue.cc:237] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.748884 20266 catalog_manager.cc:5484] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 2 to 3, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 3 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:50.864429 20661 raft_consensus.cc:979] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:50.864919 20689 raft_consensus.cc:1067] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Signalling peer c8f8d10dd45c4db99cb13d82bd75b0ab to start an election
I20240506 15:13:50.866616 20445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621"
dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
from {username='slave'} at 127.0.0.1:55092
I20240506 15:13:50.867094 20445 raft_consensus.cc:482] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:50.867362 20445 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:50.874382 20445 raft_consensus.cc:504] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.876602 20445 leader_election.cc:288] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 4 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131:42587)
I20240506 15:13:50.877588 20661 raft_consensus.cc:979] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:50.877403 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:50.877741 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6243c28ccf4b60b4f9ad7219616621" candidate_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
I20240506 15:13:50.878084 20370 raft_consensus.cc:3032] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:13:50.878340 20689 raft_consensus.cc:1067] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Signalling peer 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:50.878448 20518 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:50.878648 20370 raft_consensus.cc:729] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:50.880489 20519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:58296
I20240506 15:13:50.880283 20370 consensus_queue.cc:260] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.881155 20519 raft_consensus.cc:482] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:50.881544 20519 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.882138 20370 raft_consensus.cc:3037] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:13:50.886585 20518 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 4.
I20240506 15:13:50.888083 20413 leader_election.cc:302] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:50.889016 20715 raft_consensus.cc:2781] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:13:50.889689 20715 raft_consensus.cc:686] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 LEADER]: Becoming Leader. State: Replica: c8f8d10dd45c4db99cb13d82bd75b0ab, State: Running, Role: LEADER
I20240506 15:13:50.890731 20715 consensus_queue.cc:237] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.891935 20519 raft_consensus.cc:504] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.894320 20370 raft_consensus.cc:2454] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f8d10dd45c4db99cb13d82bd75b0ab in term 4.
I20240506 15:13:50.896447 20519 leader_election.cc:288] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 3 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:50.896340 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:50.896359 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d70f8fe2e65455895c3f33d2fc84b91" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:50.897214 20445 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.897477 20370 raft_consensus.cc:3032] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:13:50.897859 20370 raft_consensus.cc:729] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:50.898583 20370 consensus_queue.cc:260] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.900295 20370 raft_consensus.cc:3037] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:13:50.904696 20445 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 3.
I20240506 15:13:50.906116 20486 leader_election.cc:302] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:50.906987 20708 raft_consensus.cc:2781] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:13:50.906821 20266 catalog_manager.cc:5484] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab reported cstate change: term changed from 3 to 4, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130). New cstate: current_term: 4 leader_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240506 15:13:50.909668 20370 raft_consensus.cc:2454] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 3.
I20240506 15:13:50.911768 20708 raft_consensus.cc:686] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:50.912813 20708 consensus_queue.cc:237] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.922546 20266 catalog_manager.cc:5484] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 2 to 3, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:50.930081 20661 raft_consensus.cc:979] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: : Instructing follower 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:50.930563 20661 raft_consensus.cc:1067] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 LEADER]: Signalling peer 7a0a151b64f54c71a48043fe8c832de8 to start an election
I20240506 15:13:50.932338 20519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1fea309275b4b52b577085109a48795"
dest_uuid: "7a0a151b64f54c71a48043fe8c832de8"
from {username='slave'} at 127.0.0.1:58296
I20240506 15:13:50.932835 20519 raft_consensus.cc:482] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:13:50.933153 20519 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:50.939599 20519 raft_consensus.cc:504] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.941232 20519 leader_election.cc:288] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 5 election: Requested vote from peers 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331), c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899)
I20240506 15:13:50.942304 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab"
I20240506 15:13:50.942305 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fea309275b4b52b577085109a48795" candidate_uuid: "7a0a151b64f54c71a48043fe8c832de8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1a1b2df2b681456b9e3e14ab555ac89f"
I20240506 15:13:50.943053 20445 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:50.943226 20370 raft_consensus.cc:3032] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:13:50.943598 20370 raft_consensus.cc:729] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1a1b2df2b681456b9e3e14ab555ac89f, State: Running, Role: LEADER
I20240506 15:13:50.944360 20370 consensus_queue.cc:260] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.945364 20370 raft_consensus.cc:3037] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:13:50.948015 20445 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 5.
I20240506 15:13:50.949040 20486 leader_election.cc:302] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0a151b64f54c71a48043fe8c832de8, c8f8d10dd45c4db99cb13d82bd75b0ab; no voters:
I20240506 15:13:50.949883 20708 raft_consensus.cc:2781] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:13:50.950376 20708 raft_consensus.cc:686] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 5 LEADER]: Becoming Leader. State: Replica: 7a0a151b64f54c71a48043fe8c832de8, State: Running, Role: LEADER
I20240506 15:13:50.950744 20370 raft_consensus.cc:2454] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0a151b64f54c71a48043fe8c832de8 in term 5.
I20240506 15:13:50.951172 20708 consensus_queue.cc:237] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } }
I20240506 15:13:50.958196 20266 catalog_manager.cc:5484] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 reported cstate change: term changed from 4 to 5, leader changed from 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129) to 7a0a151b64f54c71a48043fe8c832de8 (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7a0a151b64f54c71a48043fe8c832de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240506 15:13:51.202356 20519 raft_consensus.cc:1261] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:51.203804 20715 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:51.217166 20370 raft_consensus.cc:1261] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:51.218837 20715 consensus_queue.cc:1035] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:13:51.342103 20370 raft_consensus.cc:1261] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:51.343516 20708 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:51.356334 20445 raft_consensus.cc:1261] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:13:51.357558 20723 consensus_queue.cc:1035] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:13:51.368599 20519 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:51.369874 20715 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a151b64f54c71a48043fe8c832de8" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:51.379894 20370 raft_consensus.cc:1261] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Refusing update from remote peer c8f8d10dd45c4db99cb13d82bd75b0ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:13:51.381505 20714 consensus_queue.cc:1035] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:13:51.386433 20371 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 5 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:13:51.387924 20723 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1b2df2b681456b9e3e14ab555ac89f" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:13:51.396881 20445 raft_consensus.cc:1261] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 5 FOLLOWER]: Refusing update from remote peer 7a0a151b64f54c71a48043fe8c832de8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:13:51.398586 20708 consensus_queue.cc:1035] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f8d10dd45c4db99cb13d82bd75b0ab" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:13:51.692231 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:51.700852 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:51.701364 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:52.702252 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:52.708680 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:52.709112 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:53.710265 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:53.716485 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:53.717041 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:54.718132 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:54.724709 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:54.725157 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:55.726042 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:55.732216 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:55.732652 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:56.733690 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:56.740746 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:56.741164 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:57.741976 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:57.747314 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:57.747749 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:58.748642 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:58.756944 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:58.757489 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:13:59.758330 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:13:59.763953 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240506 15:13:59.764406 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:14:00.795922 19726 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240506 15:14:00.796465 19726 tablet_server.cc:178] TabletServer@127.19.67.129:0 shutting down...
I20240506 15:14:00.835896 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:14:00.836973 19726 tablet_replica.cc:330] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.837978 19726 raft_consensus.cc:2229] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.839167 19726 raft_consensus.cc:2258] T ac3f567b0fef40059673b90d4899f471 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.841533 19726 tablet_replica.cc:330] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.842152 19726 raft_consensus.cc:2229] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.842700 19726 raft_consensus.cc:2258] T eda181b266b74a7388b5a86923942cb2 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.845075 19726 tablet_replica.cc:330] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.845783 19726 raft_consensus.cc:2229] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.846249 19726 raft_consensus.cc:2258] T 7e41659303f74c5084ff0a5d2ae0ac04 P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.848889 19726 tablet_replica.cc:330] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.849654 19726 raft_consensus.cc:2229] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.850289 19726 raft_consensus.cc:2258] T 5d70f8fe2e65455895c3f33d2fc84b91 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.853286 19726 tablet_replica.cc:330] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.854262 19726 raft_consensus.cc:2229] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 LEADER]: Raft consensus shutting down.
W20240506 15:14:00.854653 20411 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20240506 15:14:00.855782 19726 raft_consensus.cc:2258] T da6e40f376fc4c8f8ff453ea1830a594 P 1a1b2df2b681456b9e3e14ab555ac89f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.858703 19726 tablet_replica.cc:330] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
W20240506 15:14:00.859069 20411 consensus_peers.cc:485] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab -> Peer 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331): Couldn't send request to peer 1a1b2df2b681456b9e3e14ab555ac89f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:00.859421 19726 raft_consensus.cc:2229] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 5 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.859934 19726 raft_consensus.cc:2258] T d1fea309275b4b52b577085109a48795 P 1a1b2df2b681456b9e3e14ab555ac89f [term 5 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.861879 19726 tablet_replica.cc:330] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.862413 19726 raft_consensus.cc:2229] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.862859 19726 raft_consensus.cc:2258] T 7c6243c28ccf4b60b4f9ad7219616621 P 1a1b2df2b681456b9e3e14ab555ac89f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.865101 19726 tablet_replica.cc:330] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.866009 19726 raft_consensus.cc:2229] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.866602 19726 raft_consensus.cc:2258] T a07f94eb5f7a4502a4ac643e466d4cf9 P 1a1b2df2b681456b9e3e14ab555ac89f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.868824 19726 tablet_replica.cc:330] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.869364 19726 raft_consensus.cc:2229] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.870231 19726 raft_consensus.cc:2258] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.872619 19726 tablet_replica.cc:330] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f: stopping tablet replica
I20240506 15:14:00.873220 19726 raft_consensus.cc:2229] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.874177 19726 raft_consensus.cc:2258] T e6c21597f811461ba94aac038acf0a0b P 1a1b2df2b681456b9e3e14ab555ac89f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.905727 19726 tablet_server.cc:194] TabletServer@127.19.67.129:0 shutdown complete.
W20240506 15:14:00.905727 20485 consensus_peers.cc:485] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 -> Peer 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331): Couldn't send request to peer 1a1b2df2b681456b9e3e14ab555ac89f. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.129:34331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:00.927079 19726 tablet_server.cc:178] TabletServer@127.19.67.130:0 shutting down...
I20240506 15:14:00.961315 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:14:00.961943 19726 tablet_replica.cc:330] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.962561 19726 raft_consensus.cc:2229] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.963450 19726 raft_consensus.cc:2258] T a07f94eb5f7a4502a4ac643e466d4cf9 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.965379 19726 tablet_replica.cc:330] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.965917 19726 raft_consensus.cc:2229] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 5 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.966341 19726 raft_consensus.cc:2258] T d1fea309275b4b52b577085109a48795 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 5 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.968024 19726 tablet_replica.cc:330] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.968626 19726 raft_consensus.cc:2229] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.969135 19726 raft_consensus.cc:2258] T da6e40f376fc4c8f8ff453ea1830a594 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.972049 19726 tablet_replica.cc:330] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.972855 19726 raft_consensus.cc:2229] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.973309 19726 raft_consensus.cc:2258] T ac3f567b0fef40059673b90d4899f471 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus is shut down!
W20240506 15:14:00.973625 20411 consensus_peers.cc:485] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab -> Peer 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331): Couldn't send request to peer 1a1b2df2b681456b9e3e14ab555ac89f. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.129:34331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:00.975069 19726 tablet_replica.cc:330] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.975577 19726 raft_consensus.cc:2229] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.975955 19726 raft_consensus.cc:2258] T 6f4dc049405b4c75a4e8cf154ae8f7fa P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.977788 19726 tablet_replica.cc:330] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.978261 19726 raft_consensus.cc:2229] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.978721 19726 raft_consensus.cc:2258] T e6c21597f811461ba94aac038acf0a0b P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.980418 19726 tablet_replica.cc:330] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.981174 19726 raft_consensus.cc:2229] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.982097 19726 raft_consensus.cc:2258] T eda181b266b74a7388b5a86923942cb2 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.983881 19726 tablet_replica.cc:330] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.984459 19726 raft_consensus.cc:2229] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 LEADER]: Raft consensus shutting down.
I20240506 15:14:00.985158 19726 raft_consensus.cc:2258] T 7c6243c28ccf4b60b4f9ad7219616621 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.986909 19726 tablet_replica.cc:330] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.987342 19726 raft_consensus.cc:2229] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.987723 19726 raft_consensus.cc:2258] T 7e41659303f74c5084ff0a5d2ae0ac04 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:00.989436 19726 tablet_replica.cc:330] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab: stopping tablet replica
I20240506 15:14:00.989847 19726 raft_consensus.cc:2229] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:00.990229 19726 raft_consensus.cc:2258] T 5d70f8fe2e65455895c3f33d2fc84b91 P c8f8d10dd45c4db99cb13d82bd75b0ab [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.021592 19726 tablet_server.cc:194] TabletServer@127.19.67.130:0 shutdown complete.
W20240506 15:14:01.044611 20485 consensus_peers.cc:485] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 -> Peer 1a1b2df2b681456b9e3e14ab555ac89f (127.19.67.129:34331): Couldn't send request to peer 1a1b2df2b681456b9e3e14ab555ac89f. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.129:34331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:01.046666 19726 tablet_server.cc:178] TabletServer@127.19.67.131:0 shutting down...
W20240506 15:14:01.055186 20486 consensus_peers.cc:485] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 -> Peer c8f8d10dd45c4db99cb13d82bd75b0ab (127.19.67.130:32899): Couldn't send request to peer c8f8d10dd45c4db99cb13d82bd75b0ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.67.130:32899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:01.078994 19726 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240506 15:14:01.079738 19726 tablet_replica.cc:330] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.080405 19726 raft_consensus.cc:2229] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 5 LEADER]: Raft consensus shutting down.
I20240506 15:14:01.081228 19726 raft_consensus.cc:2258] T d1fea309275b4b52b577085109a48795 P 7a0a151b64f54c71a48043fe8c832de8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.083206 19726 tablet_replica.cc:330] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.083689 19726 raft_consensus.cc:2229] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.084043 19726 raft_consensus.cc:2258] T a07f94eb5f7a4502a4ac643e466d4cf9 P 7a0a151b64f54c71a48043fe8c832de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.085750 19726 tablet_replica.cc:330] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.086220 19726 raft_consensus.cc:2229] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.086586 19726 raft_consensus.cc:2258] T 6f4dc049405b4c75a4e8cf154ae8f7fa P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.088104 19726 tablet_replica.cc:330] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.088615 19726 raft_consensus.cc:2229] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.088990 19726 raft_consensus.cc:2258] T e6c21597f811461ba94aac038acf0a0b P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.090445 19726 tablet_replica.cc:330] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.091203 19726 raft_consensus.cc:2229] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 LEADER]: Raft consensus shutting down.
I20240506 15:14:01.091970 19726 raft_consensus.cc:2258] T 7e41659303f74c5084ff0a5d2ae0ac04 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.094192 19726 tablet_replica.cc:330] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.095000 19726 raft_consensus.cc:2229] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 LEADER]: Raft consensus shutting down.
I20240506 15:14:01.095892 19726 raft_consensus.cc:2258] T 5d70f8fe2e65455895c3f33d2fc84b91 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.097724 19726 tablet_replica.cc:330] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.098306 19726 raft_consensus.cc:2229] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.098901 19726 raft_consensus.cc:2258] T ac3f567b0fef40059673b90d4899f471 P 7a0a151b64f54c71a48043fe8c832de8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.100768 19726 tablet_replica.cc:330] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.101297 19726 raft_consensus.cc:2229] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.101725 19726 raft_consensus.cc:2258] T da6e40f376fc4c8f8ff453ea1830a594 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.103413 19726 tablet_replica.cc:330] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.103905 19726 raft_consensus.cc:2229] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.104375 19726 raft_consensus.cc:2258] T eda181b266b74a7388b5a86923942cb2 P 7a0a151b64f54c71a48043fe8c832de8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.106007 19726 tablet_replica.cc:330] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8: stopping tablet replica
I20240506 15:14:01.106437 19726 raft_consensus.cc:2229] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:01.106894 19726 raft_consensus.cc:2258] T 7c6243c28ccf4b60b4f9ad7219616621 P 7a0a151b64f54c71a48043fe8c832de8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.138430 19726 tablet_server.cc:194] TabletServer@127.19.67.131:0 shutdown complete.
I20240506 15:14:01.157054 19726 master.cc:537] Master@127.19.67.190:35243 shutting down...
I20240506 15:14:01.182931 19726 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 1 LEADER]: Raft consensus shutting down.
I20240506 15:14:01.183454 19726 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:01.183738 19726 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1d4b7e87dc814e58a45f3a243f6d9657: stopping tablet replica
I20240506 15:14:01.204313 19726 master.cc:552] Master@127.19.67.190:35243 shutdown complete.
1a1b2df2b681456b9e3e14ab555ac89f 5
7a0a151b64f54c71a48043fe8c832de8 1
c8f8d10dd45c4db99cb13d82bd75b0ab 4
1a1b2df2b681456b9e3e14ab555ac89f 3
7a0a151b64f54c71a48043fe8c832de8 3
c8f8d10dd45c4db99cb13d82bd75b0ab 4
1a1b2df2b681456b9e3e14ab555ac89f 8
7a0a151b64f54c71a48043fe8c832de8 1
c8f8d10dd45c4db99cb13d82bd75b0ab 1
1a1b2df2b681456b9e3e14ab555ac89f 4
7a0a151b64f54c71a48043fe8c832de8 3
c8f8d10dd45c4db99cb13d82bd75b0ab 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28827 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240506 15:14:01.264461 19726 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.67.190:37083
I20240506 15:14:01.265625 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:14:01.272632 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.274520 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.274720 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.276994 19726 server_base.cc:1034] running on GCE node
I20240506 15:14:01.278445 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:14:01.278729 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:14:01.278901 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008441278882 us; error 0 us; skew 500 ppm
I20240506 15:14:01.279429 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:14:01.282765 19726 webserver.cc:460] Webserver started at http://127.19.67.190:33329/ using document root <none> and password file <none>
I20240506 15:14:01.283274 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:14:01.283466 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:14:01.283751 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:14:01.285028 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/master-0-root/instance:
uuid: "785bf14009d340bcbb6be5e678b5f9cf"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.290014 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240506 15:14:01.294010 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.295037 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240506 15:14:01.295377 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/master-0-root
uuid: "785bf14009d340bcbb6be5e678b5f9cf"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.295701 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:14:01.319326 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:14:01.320868 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:14:01.366869 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.190:37083
I20240506 15:14:01.366983 20814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.190:37083 every 8 connection(s)
I20240506 15:14:01.371390 20815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:01.382393 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Bootstrap starting.
I20240506 15:14:01.386543 20815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:01.391373 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: No bootstrap required, opened a new log
I20240506 15:14:01.393455 20815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER }
I20240506 15:14:01.394624 20815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:01.394866 20815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785bf14009d340bcbb6be5e678b5f9cf, State: Initialized, Role: FOLLOWER
I20240506 15:14:01.395493 20815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER }
I20240506 15:14:01.396389 20815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 15:14:01.397017 20815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 15:14:01.397482 20815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:01.401983 20815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER }
I20240506 15:14:01.402545 20815 leader_election.cc:302] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 785bf14009d340bcbb6be5e678b5f9cf; no voters:
I20240506 15:14:01.403694 20815 leader_election.cc:288] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 15:14:01.404136 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:01.405568 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [term 1 LEADER]: Becoming Leader. State: Replica: 785bf14009d340bcbb6be5e678b5f9cf, State: Running, Role: LEADER
I20240506 15:14:01.406330 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER }
I20240506 15:14:01.406701 20815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [sys.catalog]: configured and running, proceeding with master startup.
I20240506 15:14:01.412488 20820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 785bf14009d340bcbb6be5e678b5f9cf. Latest consensus state: current_term: 1 leader_uuid: "785bf14009d340bcbb6be5e678b5f9cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER } }
I20240506 15:14:01.413041 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "785bf14009d340bcbb6be5e678b5f9cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785bf14009d340bcbb6be5e678b5f9cf" member_type: VOTER } }
I20240506 15:14:01.414106 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [sys.catalog]: This master's current role is: LEADER
I20240506 15:14:01.414009 20820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf [sys.catalog]: This master's current role is: LEADER
I20240506 15:14:01.416352 20826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 15:14:01.422657 20826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 15:14:01.424595 19726 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240506 15:14:01.432013 20826 catalog_manager.cc:1347] Generated new cluster ID: dabe3b5e15514a54b219157419cd3709
I20240506 15:14:01.432423 20826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 15:14:01.463376 20826 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 15:14:01.464721 20826 catalog_manager.cc:1504] Loading token signing keys...
I20240506 15:14:01.481571 20826 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Generated new TSK 0
I20240506 15:14:01.482199 20826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 15:14:01.491801 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:14:01.500005 20836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.502614 20837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.503332 20839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.504887 19726 server_base.cc:1034] running on GCE node
I20240506 15:14:01.505736 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:14:01.505928 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:14:01.506053 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008441506042 us; error 0 us; skew 500 ppm
I20240506 15:14:01.506569 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:14:01.509368 19726 webserver.cc:460] Webserver started at http://127.19.67.129:33559/ using document root <none> and password file <none>
I20240506 15:14:01.509856 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:14:01.510026 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:14:01.510246 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:14:01.511384 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/instance:
uuid: "00fe71c1675441379f5f2c7490f9e63a"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.517839 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240506 15:14:01.522048 20844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.523031 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240506 15:14:01.523448 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root
uuid: "00fe71c1675441379f5f2c7490f9e63a"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.523784 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:14:01.548417 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:14:01.549940 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:14:01.552037 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:14:01.554699 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:14:01.554912 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.555172 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:14:01.555349 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.633598 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.129:44145
I20240506 15:14:01.633725 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.129:44145 every 8 connection(s)
I20240506 15:14:01.641453 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:14:01.650177 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.654331 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:01.656057 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.656312 19726 server_base.cc:1034] running on GCE node
I20240506 15:14:01.657944 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:14:01.658227 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:14:01.658641 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008441658606 us; error 0 us; skew 500 ppm
I20240506 15:14:01.658983 20908 heartbeater.cc:347] Connected to a master server at 127.19.67.190:37083
I20240506 15:14:01.659415 20908 heartbeater.cc:464] Registering TS with master...
I20240506 15:14:01.659539 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:14:01.660245 20908 heartbeater.cc:511] Master 127.19.67.190:37083 requested a full tablet report, sending...
I20240506 15:14:01.662922 19726 webserver.cc:460] Webserver started at http://127.19.67.130:41337/ using document root <none> and password file <none>
I20240506 15:14:01.663367 20780 ts_manager.cc:194] Registered new tserver with Master: 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:01.663563 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:14:01.663857 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:14:01.664220 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:14:01.665851 20780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54780
I20240506 15:14:01.665920 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/instance:
uuid: "ecededcd854842d4867fda9fa63a4b66"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.672022 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 15:14:01.676548 20920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.677736 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240506 15:14:01.678037 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root
uuid: "ecededcd854842d4867fda9fa63a4b66"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.678344 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:14:01.709615 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:14:01.710968 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:14:01.712764 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:14:01.715773 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:14:01.716061 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.716400 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:14:01.716579 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.766957 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.130:40545
I20240506 15:14:01.767081 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.130:40545 every 8 connection(s)
I20240506 15:14:01.772866 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 15:14:01.786573 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.787230 20983 heartbeater.cc:347] Connected to a master server at 127.19.67.190:37083
I20240506 15:14:01.787652 20983 heartbeater.cc:464] Registering TS with master...
I20240506 15:14:01.788574 20983 heartbeater.cc:511] Master 127.19.67.190:37083 requested a full tablet report, sending...
W20240506 15:14:01.789579 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.789790 19726 server_base.cc:1034] running on GCE node
I20240506 15:14:01.790822 20780 ts_manager.cc:194] Registered new tserver with Master: ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:01.792440 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:01.792495 20780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54782
I20240506 15:14:01.793964 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:14:01.794234 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:14:01.794448 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008441794430 us; error 0 us; skew 500 ppm
I20240506 15:14:01.795552 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:14:01.798676 19726 webserver.cc:460] Webserver started at http://127.19.67.131:43631/ using document root <none> and password file <none>
I20240506 15:14:01.799201 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:14:01.799378 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:14:01.799643 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:14:01.800985 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/instance:
uuid: "7610624a16b44896b6aacea1928ab20e"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.805636 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240506 15:14:01.809376 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.810190 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240506 15:14:01.810520 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root
uuid: "7610624a16b44896b6aacea1928ab20e"
format_stamp: "Formatted at 2024-05-06 15:14:01 on dist-test-slave-8gjm"
I20240506 15:14:01.810801 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:14:01.825235 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:14:01.826448 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:14:01.828012 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:14:01.830449 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:14:01.830647 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.830895 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:14:01.831046 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:01.877043 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.131:42913
I20240506 15:14:01.877089 21056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.131:42913 every 8 connection(s)
I20240506 15:14:01.891737 21057 heartbeater.cc:347] Connected to a master server at 127.19.67.190:37083
I20240506 15:14:01.892079 21057 heartbeater.cc:464] Registering TS with master...
I20240506 15:14:01.892844 21057 heartbeater.cc:511] Master 127.19.67.190:37083 requested a full tablet report, sending...
I20240506 15:14:01.894776 20780 ts_manager.cc:194] Registered new tserver with Master: 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:01.895745 19726 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014770259s
I20240506 15:14:01.896405 20780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54786
I20240506 15:14:01.898909 19726 test_util.cc:268] Using random seed: -2064164985
I20240506 15:14:01.928805 20780 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54788:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 15:14:01.935600 20780 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 15:14:02.165356 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 815d66b0c2c24db2aa336b62f17837a1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240506 15:14:02.167050 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d66b0c2c24db2aa336b62f17837a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.167344 20943 tablet_service.cc:1467] Processing CreateTablet for tablet e81c09e0400045b180379648eed9b219 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240506 15:14:02.168766 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81c09e0400045b180379648eed9b219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.169644 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 629a891c5ba546aeb0e0027a02f1b955 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240506 15:14:02.170070 20942 tablet_service.cc:1467] Processing CreateTablet for tablet 8884787c4c4b474685c7e903e8789e90 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240506 15:14:02.171212 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629a891c5ba546aeb0e0027a02f1b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.165242 20948 tablet_service.cc:1467] Processing CreateTablet for tablet b96eaaaebde34390bee0dafdfb156080 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240506 15:14:02.173074 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884787c4c4b474685c7e903e8789e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.173653 20944 tablet_service.cc:1467] Processing CreateTablet for tablet c12c136a9dc4474897d6b8a669145139 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240506 15:14:02.174644 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 0693482e35a64dfc90d3944c134718da (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240506 15:14:02.175215 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c136a9dc4474897d6b8a669145139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.176000 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0693482e35a64dfc90d3944c134718da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.179090 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96eaaaebde34390bee0dafdfb156080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.178890 21022 tablet_service.cc:1467] Processing CreateTablet for tablet b96eaaaebde34390bee0dafdfb156080 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240506 15:14:02.183920 20940 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0fa8a9db084a51a74caa1382efc1c5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240506 15:14:02.185635 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0fa8a9db084a51a74caa1382efc1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.185905 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96eaaaebde34390bee0dafdfb156080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.189546 20939 tablet_service.cc:1467] Processing CreateTablet for tablet e7268b79c9094189823e8d245b8c579a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240506 15:14:02.191113 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7268b79c9094189823e8d245b8c579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.179111 21021 tablet_service.cc:1467] Processing CreateTablet for tablet 815d66b0c2c24db2aa336b62f17837a1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240506 15:14:02.193583 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d66b0c2c24db2aa336b62f17837a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.180944 20941 tablet_service.cc:1467] Processing CreateTablet for tablet 7e60758583b247c2946eb30ea2681286 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240506 15:14:02.197767 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e60758583b247c2946eb30ea2681286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.226167 21020 tablet_service.cc:1467] Processing CreateTablet for tablet 629a891c5ba546aeb0e0027a02f1b955 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240506 15:14:02.227699 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629a891c5ba546aeb0e0027a02f1b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.229518 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 0693482e35a64dfc90d3944c134718da (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240506 15:14:02.230922 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0693482e35a64dfc90d3944c134718da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.234220 20873 tablet_service.cc:1467] Processing CreateTablet for tablet b96eaaaebde34390bee0dafdfb156080 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240506 15:14:02.235762 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96eaaaebde34390bee0dafdfb156080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.237493 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 629a891c5ba546aeb0e0027a02f1b955 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240506 15:14:02.239002 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629a891c5ba546aeb0e0027a02f1b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.241114 21015 tablet_service.cc:1467] Processing CreateTablet for tablet 7e60758583b247c2946eb30ea2681286 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240506 15:14:02.241446 20869 tablet_service.cc:1467] Processing CreateTablet for tablet c12c136a9dc4474897d6b8a669145139 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240506 15:14:02.242637 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e60758583b247c2946eb30ea2681286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.242730 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c136a9dc4474897d6b8a669145139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.245488 21016 tablet_service.cc:1467] Processing CreateTablet for tablet 8884787c4c4b474685c7e903e8789e90 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240506 15:14:02.247119 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884787c4c4b474685c7e903e8789e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.245488 20868 tablet_service.cc:1467] Processing CreateTablet for tablet e81c09e0400045b180379648eed9b219 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240506 15:14:02.237493 21013 tablet_service.cc:1467] Processing CreateTablet for tablet e7268b79c9094189823e8d245b8c579a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240506 15:14:02.249176 21017 tablet_service.cc:1467] Processing CreateTablet for tablet e81c09e0400045b180379648eed9b219 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240506 15:14:02.249787 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7268b79c9094189823e8d245b8c579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.250622 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81c09e0400045b180379648eed9b219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.236022 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 815d66b0c2c24db2aa336b62f17837a1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240506 15:14:02.251869 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d66b0c2c24db2aa336b62f17837a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.238616 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0fa8a9db084a51a74caa1382efc1c5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240506 15:14:02.252769 21018 tablet_service.cc:1467] Processing CreateTablet for tablet c12c136a9dc4474897d6b8a669145139 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240506 15:14:02.253557 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0fa8a9db084a51a74caa1382efc1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.254633 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c136a9dc4474897d6b8a669145139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.246644 20870 tablet_service.cc:1467] Processing CreateTablet for tablet 0693482e35a64dfc90d3944c134718da (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240506 15:14:02.256742 20868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81c09e0400045b180379648eed9b219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.258507 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0693482e35a64dfc90d3944c134718da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.260999 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 8884787c4c4b474685c7e903e8789e90 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240506 15:14:02.262516 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884787c4c4b474685c7e903e8789e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.266079 20866 tablet_service.cc:1467] Processing CreateTablet for tablet 7e60758583b247c2946eb30ea2681286 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240506 15:14:02.267606 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e60758583b247c2946eb30ea2681286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.269726 20865 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0fa8a9db084a51a74caa1382efc1c5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240506 15:14:02.271664 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0fa8a9db084a51a74caa1382efc1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.273547 20863 tablet_service.cc:1467] Processing CreateTablet for tablet e7268b79c9094189823e8d245b8c579a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240506 15:14:02.274930 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7268b79c9094189823e8d245b8c579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.322104 20945 tablet_service.cc:1467] Processing CreateTablet for tablet f207fe580fd74c83b4540a552300d5b8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240506 15:14:02.323473 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f207fe580fd74c83b4540a552300d5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.338168 20943 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7c34467d5a4c759c46b85eccf0d47d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240506 15:14:02.347398 20871 tablet_service.cc:1467] Processing CreateTablet for tablet f207fe580fd74c83b4540a552300d5b8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240506 15:14:02.363869 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f207fe580fd74c83b4540a552300d5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.365610 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 801b36bb14184241921c1b97377314fb (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240506 15:14:02.367029 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801b36bb14184241921c1b97377314fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.369632 21072 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:02.379087 21019 tablet_service.cc:1467] Processing CreateTablet for tablet f207fe580fd74c83b4540a552300d5b8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240506 15:14:02.379638 20944 tablet_service.cc:1467] Processing CreateTablet for tablet 48f44ddbf0044d1c9c94508ba5aa2406 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240506 15:14:02.380506 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f207fe580fd74c83b4540a552300d5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.378487 20945 tablet_service.cc:1467] Processing CreateTablet for tablet a9f4d07ea30646d0805b63bb641358be (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240506 15:14:02.380568 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7c34467d5a4c759c46b85eccf0d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.402774 21072 tablet_bootstrap.cc:654] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:02.409307 20940 tablet_service.cc:1467] Processing CreateTablet for tablet 481a6073b18543d58315ee2fe4522232 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240506 15:14:02.409965 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 23b7301301694836b2d08bf22e3f2e38 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240506 15:14:02.411301 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b7301301694836b2d08bf22e3f2e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.411634 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f4d07ea30646d0805b63bb641358be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.412570 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f44ddbf0044d1c9c94508ba5aa2406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.422467 21022 tablet_service.cc:1467] Processing CreateTablet for tablet 801b36bb14184241921c1b97377314fb (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240506 15:14:02.422654 21021 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7c34467d5a4c759c46b85eccf0d47d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
W20240506 15:14:02.423426 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:02.423900 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801b36bb14184241921c1b97377314fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.433436 21018 tablet_service.cc:1467] Processing CreateTablet for tablet a9f4d07ea30646d0805b63bb641358be (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240506 15:14:02.434840 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f4d07ea30646d0805b63bb641358be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.436911 20948 tablet_service.cc:1467] Processing CreateTablet for tablet 23b7301301694836b2d08bf22e3f2e38 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240506 15:14:02.439356 20941 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240506 15:14:02.424633 21015 tablet_service.cc:1467] Processing CreateTablet for tablet 23b7301301694836b2d08bf22e3f2e38 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240506 15:14:02.441228 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b7301301694836b2d08bf22e3f2e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.441524 20868 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7c34467d5a4c759c46b85eccf0d47d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240506 15:14:02.442823 20868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7c34467d5a4c759c46b85eccf0d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.444603 21073 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:02.444444 20865 tablet_service.cc:1467] Processing CreateTablet for tablet 801b36bb14184241921c1b97377314fb (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240506 15:14:02.445739 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801b36bb14184241921c1b97377314fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.448879 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca08769950c4ef2bae0df854c025154 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240506 15:14:02.449339 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 48f44ddbf0044d1c9c94508ba5aa2406 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240506 15:14:02.450338 20939 tablet_service.cc:1467] Processing CreateTablet for tablet f5a9099e10624a769e45882e9b864930 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240506 15:14:02.384137 20942 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ce6e0da3e465f875171ac339f3b50 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240506 15:14:02.453199 20867 tablet_service.cc:1467] Processing CreateTablet for tablet a9f4d07ea30646d0805b63bb641358be (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240506 15:14:02.454599 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f4d07ea30646d0805b63bb641358be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.450752 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f44ddbf0044d1c9c94508ba5aa2406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.457468 20944 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcd32ac1e314101bae9c062a76798df (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240506 15:14:02.459646 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240506 15:14:02.460839 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 48f44ddbf0044d1c9c94508ba5aa2406 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240506 15:14:02.460321 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7c34467d5a4c759c46b85eccf0d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.469511 20870 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ce6e0da3e465f875171ac339f3b50 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240506 15:14:02.465490 21017 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ce6e0da3e465f875171ac339f3b50 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240506 15:14:02.477427 20873 tablet_service.cc:1467] Processing CreateTablet for tablet 481a6073b18543d58315ee2fe4522232 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240506 15:14:02.479267 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481a6073b18543d58315ee2fe4522232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.465212 21074 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:02.488782 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcd32ac1e314101bae9c062a76798df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.489492 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ce6e0da3e465f875171ac339f3b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.492734 20863 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240506 15:14:02.494304 21020 tablet_service.cc:1467] Processing CreateTablet for tablet 481a6073b18543d58315ee2fe4522232 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240506 15:14:02.495606 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481a6073b18543d58315ee2fe4522232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.496584 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ce6e0da3e465f875171ac339f3b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.497257 21013 tablet_service.cc:1467] Processing CreateTablet for tablet f5a9099e10624a769e45882e9b864930 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240506 15:14:02.499562 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a9099e10624a769e45882e9b864930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.505201 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a9099e10624a769e45882e9b864930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.507864 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240506 15:14:02.509869 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.482057 20869 tablet_service.cc:1467] Processing CreateTablet for tablet f5a9099e10624a769e45882e9b864930 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240506 15:14:02.486569 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f44ddbf0044d1c9c94508ba5aa2406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.512357 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a9099e10624a769e45882e9b864930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.512997 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481a6073b18543d58315ee2fe4522232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.516570 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ce6e0da3e465f875171ac339f3b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.488816 21072 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:02.524945 21072 ts_tablet_manager.cc:1367] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.156s user 0.006s sys 0.006s
I20240506 15:14:02.525933 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b7301301694836b2d08bf22e3f2e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.528591 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.532625 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca08769950c4ef2bae0df854c025154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.537297 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.538115 21074 tablet_bootstrap.cc:654] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:02.513016 20866 tablet_service.cc:1467] Processing CreateTablet for tablet b51ffbf114654032ba2038e989eec2bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240506 15:14:02.546717 21073 tablet_bootstrap.cc:654] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:02.544606 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea5e32ab06245dfb58c1f9b00c7fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.549793 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51ffbf114654032ba2038e989eec2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.552817 21016 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca08769950c4ef2bae0df854c025154 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240506 15:14:02.554258 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca08769950c4ef2bae0df854c025154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.535758 21072 raft_consensus.cc:348] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.565313 21072 raft_consensus.cc:374] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:02.565711 21072 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:02.566709 21072 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.544150 20943 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54b354e91941efb28b61b312923d41 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240506 15:14:02.579201 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54b354e91941efb28b61b312923d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.587208 21074 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:02.587564 20865 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca08769950c4ef2bae0df854c025154 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240506 15:14:02.588013 21074 ts_tablet_manager.cc:1367] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.123s user 0.016s sys 0.000s
I20240506 15:14:02.604367 21074 raft_consensus.cc:348] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.604806 21017 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54b354e91941efb28b61b312923d41 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240506 15:14:02.607614 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54b354e91941efb28b61b312923d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.608644 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54b354e91941efb28b61b312923d41 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240506 15:14:02.614310 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54b354e91941efb28b61b312923d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.613956 20873 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcd32ac1e314101bae9c062a76798df (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240506 15:14:02.615564 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcd32ac1e314101bae9c062a76798df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.622465 21015 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcd32ac1e314101bae9c062a76798df (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240506 15:14:02.623886 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcd32ac1e314101bae9c062a76798df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.611732 21074 raft_consensus.cc:374] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:02.645069 21057 heartbeater.cc:503] Master 127.19.67.190:37083 was elected leader, sending a full tablet report...
I20240506 15:14:02.644927 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240506 15:14:02.649508 20868 tablet_service.cc:1467] Processing CreateTablet for tablet 1a88ec9e65b24b64be49384639f4f9f9 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240506 15:14:02.633299 21074 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:02.652920 21074 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.639959 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca08769950c4ef2bae0df854c025154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.658341 21073 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:02.658833 21073 ts_tablet_manager.cc:1367] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.239s user 0.006s sys 0.008s
I20240506 15:14:02.659585 20945 tablet_service.cc:1467] Processing CreateTablet for tablet b51ffbf114654032ba2038e989eec2bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240506 15:14:02.660601 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.661192 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51ffbf114654032ba2038e989eec2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.661906 21073 raft_consensus.cc:348] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.662678 21073 raft_consensus.cc:374] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:02.662986 21073 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:02.665598 21022 tablet_service.cc:1467] Processing CreateTablet for tablet b51ffbf114654032ba2038e989eec2bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240506 15:14:02.658176 21018 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240506 15:14:02.670595 20908 heartbeater.cc:503] Master 127.19.67.190:37083 was elected leader, sending a full tablet report...
I20240506 15:14:02.668534 21077 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:02.677301 20868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a88ec9e65b24b64be49384639f4f9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.663797 21073 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.690461 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.696069 20869 tablet_service.cc:1467] Processing CreateTablet for tablet b540a7832fba49b5b2623616240c2711 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240506 15:14:02.698168 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b540a7832fba49b5b2623616240c2711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.700153 20944 tablet_service.cc:1467] Processing CreateTablet for tablet 1a88ec9e65b24b64be49384639f4f9f9 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240506 15:14:02.701844 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a88ec9e65b24b64be49384639f4f9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.704149 21077 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.684222 21020 tablet_service.cc:1467] Processing CreateTablet for tablet b540a7832fba49b5b2623616240c2711 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240506 15:14:02.712052 21072 ts_tablet_manager.cc:1398] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.180s user 0.004s sys 0.008s
I20240506 15:14:02.728099 21072 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:02.733023 20948 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d9cec20d4d219e474797b7115172 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240506 15:14:02.738480 21074 ts_tablet_manager.cc:1398] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.141s user 0.019s sys 0.031s
I20240506 15:14:02.739499 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc90d9cec20d4d219e474797b7115172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.668804 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 1a88ec9e65b24b64be49384639f4f9f9 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240506 15:14:02.743643 21013 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d9cec20d4d219e474797b7115172 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240506 15:14:02.750217 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51ffbf114654032ba2038e989eec2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.710486 20942 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240506 15:14:02.714076 21021 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240506 15:14:02.757540 21077 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:02.766413 21074 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:02.709239 20866 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d9cec20d4d219e474797b7115172 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240506 15:14:02.776439 21079 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:02.769162 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc90d9cec20d4d219e474797b7115172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.793411 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 81525cec16e140db99cd0ef9f2717778 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240506 15:14:02.776974 21079 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.801260 21079 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:02.803334 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9c7ee4280049d091aea89be873b5c8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240506 15:14:02.804533 20940 tablet_service.cc:1467] Processing CreateTablet for tablet b540a7832fba49b5b2623616240c2711 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240506 15:14:02.809262 20983 heartbeater.cc:503] Master 127.19.67.190:37083 was elected leader, sending a full tablet report...
I20240506 15:14:02.722200 20870 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240506 15:14:02.824774 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81525cec16e140db99cd0ef9f2717778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.823140 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b540a7832fba49b5b2623616240c2711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.839430 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9c7ee4280049d091aea89be873b5c8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240506 15:14:02.807571 20939 tablet_service.cc:1467] Processing CreateTablet for tablet 81525cec16e140db99cd0ef9f2717778 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240506 15:14:02.850919 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.857717 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9c7ee4280049d091aea89be873b5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.868328 21073 ts_tablet_manager.cc:1398] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.209s user 0.014s sys 0.008s
I20240506 15:14:02.868063 20863 tablet_service.cc:1467] Processing CreateTablet for tablet 75831e0251ec4bec9c8f3ca6c4a047f0 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240506 15:14:02.869479 21073 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:02.870090 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 81525cec16e140db99cd0ef9f2717778 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240506 15:14:02.873468 21017 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9c7ee4280049d091aea89be873b5c8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240506 15:14:02.878242 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 75831e0251ec4bec9c8f3ca6c4a047f0 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240506 15:14:02.879523 20941 tablet_service.cc:1467] Processing CreateTablet for tablet ab1516eeb0ab46f4b471c209c399130f (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240506 15:14:02.884878 21074 tablet_bootstrap.cc:654] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:02.892496 20909 tablet.cc:2369] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 15:14:02.892455 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9c7ee4280049d091aea89be873b5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.894645 21018 tablet_service.cc:1467] Processing CreateTablet for tablet 75831e0251ec4bec9c8f3ca6c4a047f0 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240506 15:14:02.895509 21080 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:02.896426 20767 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 56179us
I20240506 15:14:02.896055 21080 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:02.901289 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75831e0251ec4bec9c8f3ca6c4a047f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.890966 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a88ec9e65b24b64be49384639f4f9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.911176 21080 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:02.935673 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b540a7832fba49b5b2623616240c2711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.952711 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.964318 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9c7ee4280049d091aea89be873b5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.990805 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75831e0251ec4bec9c8f3ca6c4a047f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.988477 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75831e0251ec4bec9c8f3ca6c4a047f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.000079 20873 tablet_service.cc:1467] Processing CreateTablet for tablet ab1516eeb0ab46f4b471c209c399130f (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240506 15:14:03.022550 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1516eeb0ab46f4b471c209c399130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:02.999473 20865 tablet_service.cc:1467] Processing CreateTablet for tablet c89eb38afe3b43719809589e0aeb7eef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240506 15:14:03.025799 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89eb38afe3b43719809589e0aeb7eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.027316 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 35b78f712a194a51abc96c8500a9a611 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240506 15:14:03.041602 20944 tablet_service.cc:1467] Processing CreateTablet for tablet c89eb38afe3b43719809589e0aeb7eef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240506 15:14:03.046865 21074 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:03.047813 21074 ts_tablet_manager.cc:1367] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.282s user 0.024s sys 0.029s
I20240506 15:14:03.060041 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b78f712a194a51abc96c8500a9a611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.069775 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 35b78f712a194a51abc96c8500a9a611 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240506 15:14:03.004724 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81525cec16e140db99cd0ef9f2717778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.010165 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.089463 21073 tablet_bootstrap.cc:654] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.094775 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b78f712a194a51abc96c8500a9a611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.114513 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89eb38afe3b43719809589e0aeb7eef. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.111375 20765 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 54277.9us
I20240506 15:14:03.116808 21016 tablet_service.cc:1467] Processing CreateTablet for tablet ab1516eeb0ab46f4b471c209c399130f (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240506 15:14:03.121985 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 208b7cbf36514c419696de9c14606c16 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240506 15:14:03.124609 20866 tablet_service.cc:1467] Processing CreateTablet for tablet 208b7cbf36514c419696de9c14606c16 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240506 15:14:03.125059 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:03.126374 20958 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:03.128432 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:03.129616 21077 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:03.130103 21077 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:03.130446 21077 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.130841 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:03.126921 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208b7cbf36514c419696de9c14606c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.123602 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208b7cbf36514c419696de9c14606c16. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.141795 20845 leader_election.cc:341] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.143460 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:03.149596 21077 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.158118 20869 tablet_service.cc:1467] Processing CreateTablet for tablet 286345f896bc43ec975183896aee147c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240506 15:14:03.159951 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286345f896bc43ec975183896aee147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.159752 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:03.160611 20958 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:03.165027 20998 leader_election.cc:341] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.136835 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc90d9cec20d4d219e474797b7115172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.169185 20958 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:03.170493 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:03.173911 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1516eeb0ab46f4b471c209c399130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.181885 20943 tablet_service.cc:1467] Processing CreateTablet for tablet e47e3786df1a41928afd8b70061d732a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240506 15:14:03.119889 21074 raft_consensus.cc:348] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.184302 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81525cec16e140db99cd0ef9f2717778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.184794 21074 raft_consensus.cc:374] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.186836 21074 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.176060 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:03.198758 20998 leader_election.cc:341] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.207124 21072 tablet_bootstrap.cc:654] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.206925 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3241be4714e97a24f66b7c1e9c4d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240506 15:14:03.212913 20873 tablet_service.cc:1467] Processing CreateTablet for tablet d84d513d60b64cba9c6bc3f8077e6b73 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240506 15:14:03.214524 21074 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.215951 20872 tablet_service.cc:1467] Processing CreateTablet for tablet edfbe39391094dce8178f175dfb05165 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240506 15:14:03.217769 21074 ts_tablet_manager.cc:1398] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.170s user 0.012s sys 0.000s
I20240506 15:14:03.218715 21074 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:03.221987 21079 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.222496 21079 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.182015 20865 tablet_service.cc:1467] Processing CreateTablet for tablet 805cea534ce0498f89b8551ba0b7e5d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240506 15:14:03.226187 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cea534ce0498f89b8551ba0b7e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.230239 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1516eeb0ab46f4b471c209c399130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.230545 20870 tablet_service.cc:1467] Processing CreateTablet for tablet 2719acf5d3614c8bb6f5d84b1bc791de (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240506 15:14:03.233076 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2719acf5d3614c8bb6f5d84b1bc791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.237000 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:03.240931 20868 tablet_service.cc:1467] Processing CreateTablet for tablet e47e3786df1a41928afd8b70061d732a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240506 15:14:03.243865 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e3786df1a41928afd8b70061d732a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.175653 21077 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:03.262887 21077 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:03.263513 21077 raft_consensus.cc:686] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:03.264792 21077 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.266988 20948 tablet_service.cc:1467] Processing CreateTablet for tablet 286345f896bc43ec975183896aee147c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240506 15:14:03.273063 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfbe39391094dce8178f175dfb05165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.279922 21020 tablet_service.cc:1467] Processing CreateTablet for tablet c89eb38afe3b43719809589e0aeb7eef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240506 15:14:03.281603 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89eb38afe3b43719809589e0aeb7eef. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.282169 20846 leader_election.cc:341] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.282862 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:03.284346 21083 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:03.289184 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:03.290719 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:03.293164 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 122790us
I20240506 15:14:03.295302 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:03.295534 20846 leader_election.cc:341] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.296711 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286345f896bc43ec975183896aee147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.297009 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:03.298532 20845 leader_election.cc:341] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.299024 20845 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:03.300036 21083 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:03.301100 21079 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.305557 21073 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:03.306100 21073 ts_tablet_manager.cc:1367] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.437s user 0.019s sys 0.018s
I20240506 15:14:03.308931 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 208b7cbf36514c419696de9c14606c16 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240506 15:14:03.298980 20869 tablet_service.cc:1467] Processing CreateTablet for tablet 10845b6e0cc149f5a5e40605a2ea7e49 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240506 15:14:03.312786 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3241be4714e97a24f66b7c1e9c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.313655 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10845b6e0cc149f5a5e40605a2ea7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.327114 21015 tablet_service.cc:1467] Processing CreateTablet for tablet e47e3786df1a41928afd8b70061d732a (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
W20240506 15:14:03.332774 20924 leader_election.cc:341] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.333309 20924 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:03.334120 21080 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:03.336045 21074 tablet_bootstrap.cc:654] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.311841 20942 tablet_service.cc:1467] Processing CreateTablet for tablet 805cea534ce0498f89b8551ba0b7e5d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240506 15:14:03.316617 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208b7cbf36514c419696de9c14606c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.339622 20870 tablet_service.cc:1467] Processing CreateTablet for tablet f6abe388354e4b7e8f0656dda82078bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240506 15:14:03.307830 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 35b78f712a194a51abc96c8500a9a611 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240506 15:14:03.344646 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cea534ce0498f89b8551ba0b7e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.280237 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 5b648042628a4aec948d269392eb5cf5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240506 15:14:03.360863 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e3786df1a41928afd8b70061d732a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.365648 21073 raft_consensus.cc:348] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.376505 21073 raft_consensus.cc:374] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.376864 21073 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.377628 21073 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.380774 21073 ts_tablet_manager.cc:1398] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.074s user 0.007s sys 0.002s
I20240506 15:14:03.381965 21073 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:03.381829 20945 tablet_service.cc:1467] Processing CreateTablet for tablet d84d513d60b64cba9c6bc3f8077e6b73 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240506 15:14:03.382411 21017 tablet_service.cc:1467] Processing CreateTablet for tablet 805cea534ce0498f89b8551ba0b7e5d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240506 15:14:03.383327 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84d513d60b64cba9c6bc3f8077e6b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.386514 20940 tablet_service.cc:1467] Processing CreateTablet for tablet edfbe39391094dce8178f175dfb05165 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240506 15:14:03.387378 21072 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:03.387914 21072 ts_tablet_manager.cc:1367] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.660s user 0.031s sys 0.014s
I20240506 15:14:03.389652 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 2719acf5d3614c8bb6f5d84b1bc791de (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240506 15:14:03.390693 21018 tablet_service.cc:1467] Processing CreateTablet for tablet 286345f896bc43ec975183896aee147c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240506 15:14:03.392568 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cea534ce0498f89b8551ba0b7e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.392408 21072 raft_consensus.cc:348] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.395299 21072 raft_consensus.cc:374] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.395699 21072 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.395797 20944 tablet_service.cc:1467] Processing CreateTablet for tablet 5b648042628a4aec948d269392eb5cf5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240506 15:14:03.397166 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b648042628a4aec948d269392eb5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.396606 21013 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3241be4714e97a24f66b7c1e9c4d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240506 15:14:03.396643 21072 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.399338 20939 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3241be4714e97a24f66b7c1e9c4d8 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240506 15:14:03.393081 20865 tablet_service.cc:1467] Processing CreateTablet for tablet bc05b4289a334b34a5f571e608bc9191 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240506 15:14:03.400743 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3241be4714e97a24f66b7c1e9c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.398646 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286345f896bc43ec975183896aee147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.400396 20863 tablet_service.cc:1467] Processing CreateTablet for tablet dae00aa7ed8440db954bef879396844e (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240506 15:14:03.404306 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3241be4714e97a24f66b7c1e9c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.360371 20868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e3786df1a41928afd8b70061d732a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.405355 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6abe388354e4b7e8f0656dda82078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.405978 20778 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:03.408638 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b648042628a4aec948d269392eb5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.411793 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2719acf5d3614c8bb6f5d84b1bc791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.413329 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84d513d60b64cba9c6bc3f8077e6b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.420843 21021 tablet_service.cc:1467] Processing CreateTablet for tablet edfbe39391094dce8178f175dfb05165 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20240506 15:14:03.426846 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:03.428547 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfbe39391094dce8178f175dfb05165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.445953 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc05b4289a334b34a5f571e608bc9191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.447052 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae00aa7ed8440db954bef879396844e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.446965 21016 tablet_service.cc:1467] Processing CreateTablet for tablet d84d513d60b64cba9c6bc3f8077e6b73 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240506 15:14:03.424136 21073 tablet_bootstrap.cc:654] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.470759 21072 ts_tablet_manager.cc:1398] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.083s user 0.007s sys 0.003s
I20240506 15:14:03.464121 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b78f712a194a51abc96c8500a9a611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.488570 21072 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:03.507547 20871 tablet_service.cc:1467] Processing CreateTablet for tablet d85358bc4d3f42d897b4c2868c83c8ec (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240506 15:14:03.510375 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85358bc4d3f42d897b4c2868c83c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.516659 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84d513d60b64cba9c6bc3f8077e6b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.523402 21072 tablet_bootstrap.cc:654] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.524701 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfbe39391094dce8178f175dfb05165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.529471 20947 tablet_service.cc:1467] Processing CreateTablet for tablet dae00aa7ed8440db954bef879396844e (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240506 15:14:03.532630 21022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47936 (request call id 24) took 1156 ms. Trace:
I20240506 15:14:03.543103 21074 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:03.543665 21074 ts_tablet_manager.cc:1367] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.325s user 0.013s sys 0.016s
I20240506 15:14:03.544988 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 91da71d5ebf84ae7844608806c0d9a94 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240506 15:14:03.494395 20941 tablet_service.cc:1467] Processing CreateTablet for tablet f6abe388354e4b7e8f0656dda82078bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240506 15:14:03.551865 20866 tablet_service.cc:1467] Processing CreateTablet for tablet f85eeb0daa004acc898342a81ba6fd8d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240506 15:14:03.554664 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91da71d5ebf84ae7844608806c0d9a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.555701 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6abe388354e4b7e8f0656dda82078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.558101 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 2719acf5d3614c8bb6f5d84b1bc791de (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240506 15:14:03.537257 21022 rpcz_store.cc:276] 0506 15:14:02.376105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:02.664612 (+288507us) service_pool.cc:225] Handling call
0506 15:14:02.675178 (+ 10566us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:02.689202 (+ 14024us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.374400 (+685198us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.496060 (+121660us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.532554 (+ 36494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.561223 21022 tablet_service.cc:1467] Processing CreateTablet for tablet 10845b6e0cc149f5a5e40605a2ea7e49 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240506 15:14:03.562577 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10845b6e0cc149f5a5e40605a2ea7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.564626 20943 tablet_service.cc:1467] Processing CreateTablet for tablet 10845b6e0cc149f5a5e40605a2ea7e49 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240506 15:14:03.566152 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10845b6e0cc149f5a5e40605a2ea7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.566903 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 6d096daea6fa4799a608705b3dc732db (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240506 15:14:03.568603 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85eeb0daa004acc898342a81ba6fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.569537 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d096daea6fa4799a608705b3dc732db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.568449 20863 tablet_service.cc:1467] Processing CreateTablet for tablet fc2acc72bba8443fb8d836e09497cb3c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240506 15:14:03.572587 20948 tablet_service.cc:1467] Processing CreateTablet for tablet bc05b4289a334b34a5f571e608bc9191 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240506 15:14:03.574519 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc05b4289a334b34a5f571e608bc9191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.550954 20870 tablet_service.cc:1467] Processing CreateTablet for tablet 809ee3c2d683400da1f7a753fea4a4a3 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240506 15:14:03.583063 21073 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:03.589174 21016 tablet_service.cc:1467] Processing CreateTablet for tablet 5b648042628a4aec948d269392eb5cf5 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240506 15:14:03.590250 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae00aa7ed8440db954bef879396844e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.590751 21074 raft_consensus.cc:348] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.594692 21073 ts_tablet_manager.cc:1367] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.213s user 0.016s sys 0.001s
I20240506 15:14:03.599501 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2719acf5d3614c8bb6f5d84b1bc791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.619731 20943 tablet_service.cc:1467] Processing CreateTablet for tablet 91da71d5ebf84ae7844608806c0d9a94 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240506 15:14:03.602268 21073 raft_consensus.cc:348] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.621340 21073 raft_consensus.cc:374] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.621691 21073 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.622594 21073 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.604843 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2acc72bba8443fb8d836e09497cb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.625381 21073 ts_tablet_manager.cc:1398] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20240506 15:14:03.626457 21073 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:03.595369 21074 raft_consensus.cc:374] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.632691 21074 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.633656 21074 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.639807 20873 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a4c20ff49de9f94f4e07d3269ef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240506 15:14:03.643201 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b648042628a4aec948d269392eb5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.644464 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ee3c2d683400da1f7a753fea4a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.652066 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a4c20ff49de9f94f4e07d3269ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.652184 20865 tablet_service.cc:1467] Processing CreateTablet for tablet faf3a7304753470787006e0ba7cf8852 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240506 15:14:03.654270 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91da71d5ebf84ae7844608806c0d9a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.654320 20869 tablet_service.cc:1467] Processing CreateTablet for tablet 81702cae5854497d947832321f954762 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240506 15:14:03.654626 21015 tablet_service.cc:1467] Processing CreateTablet for tablet f6abe388354e4b7e8f0656dda82078bc (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240506 15:14:03.662339 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6abe388354e4b7e8f0656dda82078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.672590 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf3a7304753470787006e0ba7cf8852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.673355 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81702cae5854497d947832321f954762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.678903 20940 tablet_service.cc:1467] Processing CreateTablet for tablet d85358bc4d3f42d897b4c2868c83c8ec (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240506 15:14:03.680881 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 9178d1bf59624e7f8948c33552e74a00 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240506 15:14:03.672489 20868 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50460 (request call id 36) took 1027 ms. Trace:
I20240506 15:14:03.685513 20868 rpcz_store.cc:276] 0506 15:14:02.644968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.132643 (+487675us) service_pool.cc:225] Handling call
0506 15:14:03.260871 (+128228us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.260972 (+ 101us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.640128 (+379156us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.659814 (+ 19686us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.666258 (+ 6444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.686659 21017 tablet_service.cc:1467] Processing CreateTablet for tablet dae00aa7ed8440db954bef879396844e (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240506 15:14:03.688094 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae00aa7ed8440db954bef879396844e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.689428 21074 ts_tablet_manager.cc:1398] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.145s user 0.006s sys 0.002s
I20240506 15:14:03.696929 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9178d1bf59624e7f8948c33552e74a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.698014 21074 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:03.698796 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 74536b2ff8754c03a925c7b726433266 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240506 15:14:03.701290 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74536b2ff8754c03a925c7b726433266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.702355 21013 tablet_service.cc:1467] Processing CreateTablet for tablet bc05b4289a334b34a5f571e608bc9191 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240506 15:14:03.703256 21073 tablet_bootstrap.cc:654] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.699394 21014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47936 (request call id 34) took 1091 ms. Trace:
I20240506 15:14:03.695910 20939 tablet_service.cc:1467] Processing CreateTablet for tablet f85eeb0daa004acc898342a81ba6fd8d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240506 15:14:03.729116 21079 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.738318 21079 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.740468 21079 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.741792 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:03.742806 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:03.743798 20998 consensus_peers.cc:485] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e -> Peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Couldn't send request to peer 00fe71c1675441379f5f2c7490f9e63a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 15:14:03.744339 20846 leader_election.cc:341] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.710538 21014 rpcz_store.cc:276] 0506 15:14:02.607874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.306867 (+698993us) service_pool.cc:225] Handling call
0506 15:14:03.358048 (+ 51181us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.360591 (+ 2543us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.584670 (+224079us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.698885 (+114215us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.699332 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.745224 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85eeb0daa004acc898342a81ba6fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.745133 20845 leader_election.cc:341] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.745636 20845 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:03.747269 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 91da71d5ebf84ae7844608806c0d9a94 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240506 15:14:03.722477 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc05b4289a334b34a5f571e608bc9191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.748442 21079 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:03.712586 21080 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.749706 21080 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.752614 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91da71d5ebf84ae7844608806c0d9a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.752936 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:03.754678 20924 leader_election.cc:341] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.755339 21020 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47936 (request call id 33) took 1160 ms. Trace:
I20240506 15:14:03.733214 21074 tablet_bootstrap.cc:654] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.755745 21020 rpcz_store.cc:276] 0506 15:14:02.594500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.278535 (+684035us) service_pool.cc:225] Handling call
0506 15:14:03.280911 (+ 2376us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.281014 (+ 103us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.470566 (+189552us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.754847 (+284281us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.755278 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.736469 21018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47936 (request call id 37) took 1043 ms. Trace:
I20240506 15:14:03.757150 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:03.757846 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:03.757777 21020 tablet_service.cc:1467] Processing CreateTablet for tablet d85358bc4d3f42d897b4c2868c83c8ec (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240506 15:14:03.732841 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 6d096daea6fa4799a608705b3dc732db (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240506 15:14:03.759377 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85358bc4d3f42d897b4c2868c83c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.759596 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:03.760141 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d096daea6fa4799a608705b3dc732db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.762835 21019 tablet_service.cc:1467] Processing CreateTablet for tablet f85eeb0daa004acc898342a81ba6fd8d (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240506 15:14:03.756685 21018 rpcz_store.cc:276] 0506 15:14:02.693072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.356869 (+663797us) service_pool.cc:225] Handling call
0506 15:14:03.397639 (+ 40770us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.397744 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.641606 (+243862us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.708074 (+ 66468us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.736398 (+ 28324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.769060 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85eeb0daa004acc898342a81ba6fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.789726 21073 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:03.790289 21073 ts_tablet_manager.cc:1367] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.164s user 0.017s sys 0.007s
I20240506 15:14:03.790702 20942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35794 (request call id 38) took 1096 ms. Trace:
I20240506 15:14:03.791108 20942 rpcz_store.cc:276] 0506 15:14:02.693913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.310801 (+616888us) service_pool.cc:225] Handling call
0506 15:14:03.344070 (+ 33269us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.344203 (+ 133us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.411308 (+ 67105us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.790131 (+378823us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.790641 (+ 510us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31642}
I20240506 15:14:03.793634 20942 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a4c20ff49de9f94f4e07d3269ef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240506 15:14:03.791360 20947 tablet_service.cc:1467] Processing CreateTablet for tablet fc2acc72bba8443fb8d836e09497cb3c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240506 15:14:03.793678 21073 raft_consensus.cc:348] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.795567 21073 raft_consensus.cc:374] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.795872 21073 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.771515 21080 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.786345 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85358bc4d3f42d897b4c2868c83c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.695989 20944 tablet_service.cc:1467] Processing CreateTablet for tablet 809ee3c2d683400da1f7a753fea4a4a3 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240506 15:14:03.804991 21072 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:03.807332 21072 ts_tablet_manager.cc:1367] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.328s user 0.020s sys 0.009s
I20240506 15:14:03.806058 21080 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:03.812971 21080 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:03.813362 21080 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.808053 20945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35794 (request call id 40) took 1069 ms. Trace:
I20240506 15:14:03.824626 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ee3c2d683400da1f7a753fea4a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.822912 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 63583.4us
I20240506 15:14:03.848800 21014 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a4c20ff49de9f94f4e07d3269ef (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240506 15:14:03.850701 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a4c20ff49de9f94f4e07d3269ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.852432 21107 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.852942 21107 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.855885 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:03.856608 20883 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:03.833184 21080 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.834511 21072 raft_consensus.cc:348] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.858357 21072 raft_consensus.cc:374] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.858675 21072 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.859565 21072 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.836817 21018 tablet_service.cc:1467] Processing CreateTablet for tablet 809ee3c2d683400da1f7a753fea4a4a3 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240506 15:14:03.861274 21016 tablet_service.cc:1467] Processing CreateTablet for tablet faf3a7304753470787006e0ba7cf8852 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240506 15:14:03.863129 21072 ts_tablet_manager.cc:1398] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.052s user 0.001s sys 0.004s
I20240506 15:14:03.840355 21021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47936 (request call id 41) took 1075 ms. Trace:
I20240506 15:14:03.826738 20945 rpcz_store.cc:276] 0506 15:14:02.739050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.380934 (+641884us) service_pool.cc:225] Handling call
0506 15:14:03.382862 (+ 1928us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.382957 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.581890 (+198933us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.774488 (+192598us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.806976 (+ 32488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.857851 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:03.861258 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ee3c2d683400da1f7a753fea4a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.867631 21022 tablet_service.cc:1467] Processing CreateTablet for tablet 6d096daea6fa4799a608705b3dc732db (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240506 15:14:03.869705 21074 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:03.870457 21074 ts_tablet_manager.cc:1367] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.010s
I20240506 15:14:03.872215 21080 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.880139 21080 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:03.876701 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:03.882007 21031 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.881987 21080 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:03.883219 21080 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.876802 21072 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:03.887151 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 81702cae5854497d947832321f954762 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240506 15:14:03.896589 20941 tablet_service.cc:1467] Processing CreateTablet for tablet faf3a7304753470787006e0ba7cf8852 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240506 15:14:03.878149 21015 tablet_service.cc:1467] Processing CreateTablet for tablet fc2acc72bba8443fb8d836e09497cb3c (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240506 15:14:03.878032 21074 raft_consensus.cc:348] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.864547 21021 rpcz_store.cc:276] 0506 15:14:02.765522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.419876 (+654354us) service_pool.cc:225] Handling call
0506 15:14:03.421705 (+ 1829us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.421818 (+ 113us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.680424 (+258606us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.839797 (+159373us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.840282 (+ 485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:03.883369 21107 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.880156 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:03.889430 21077 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240506 15:14:03.899292 20921 leader_election.cc:341] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.901484 21021 tablet_service.cc:1467] Processing CreateTablet for tablet 74536b2ff8754c03a925c7b726433266 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240506 15:14:03.876698 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2acc72bba8443fb8d836e09497cb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.892500 21014 tablet_service.cc:1467] Processing CreateTablet for tablet 9178d1bf59624e7f8948c33552e74a00 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240506 15:14:03.906360 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a4c20ff49de9f94f4e07d3269ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.897601 21074 raft_consensus.cc:374] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:03.908087 21074 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:03.910681 20943 tablet_service.cc:1467] Processing CreateTablet for tablet 74536b2ff8754c03a925c7b726433266 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240506 15:14:03.910040 21074 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.798075 21073 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.896008 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 9178d1bf59624e7f8948c33552e74a00 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240506 15:14:03.894330 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:03.917198 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:03.914996 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81702cae5854497d947832321f954762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.918490 21107 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:03.919059 21107 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:03.919986 21107 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.900882 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf3a7304753470787006e0ba7cf8852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.923234 21080 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.928591 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d096daea6fa4799a608705b3dc732db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.942396 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9178d1bf59624e7f8948c33552e74a00. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:03.928468 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70725.3us
I20240506 15:14:03.944617 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74536b2ff8754c03a925c7b726433266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.944460 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:03.946302 20924 leader_election.cc:341] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.948311 21015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2acc72bba8443fb8d836e09497cb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.951494 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74536b2ff8754c03a925c7b726433266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.954048 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:03.891404 21017 tablet_service.cc:1467] Processing CreateTablet for tablet 81702cae5854497d947832321f954762 (DEFAULT_TABLE table=test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240506 15:14:03.954895 20883 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.955624 21017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81702cae5854497d947832321f954762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.917573 21073 ts_tablet_manager.cc:1398] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.127s user 0.009s sys 0.000s
I20240506 15:14:03.941506 21109 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.960599 21109 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.962725 20883 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:03.963215 21109 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.942513 21074 ts_tablet_manager.cc:1398] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.072s user 0.010s sys 0.000s
I20240506 15:14:03.963917 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:03.965184 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:03.965405 20921 leader_election.cc:341] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.965842 20882 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:03.967243 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:03.968322 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:03.969542 21109 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:03.935782 21072 tablet_bootstrap.cc:654] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:03.973363 21107 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:03.974869 21107 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:03.975239 21107 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:03.986610 21074 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:03.987558 21079 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:03.988047 21079 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:03.989482 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9178d1bf59624e7f8948c33552e74a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:03.990106 21079 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:03.992781 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:03.994496 20845 leader_election.cc:341] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.995361 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:03.996618 21080 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:03.996834 20846 leader_election.cc:341] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:03.997351 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:03.998570 21079 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:03.999536 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:04.002156 20921 leader_election.cc:341] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.003437 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf3a7304753470787006e0ba7cf8852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:04.006034 21073 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.019443 21107 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.029423 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:04.029560 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:04.030154 20883 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:04.031150 20921 leader_election.cc:341] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.031515 21074 tablet_bootstrap.cc:654] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.031886 21107 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.037341 20883 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:04.039914 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.041954 21107 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.042712 21107 raft_consensus.cc:686] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:04.045964 21107 consensus_queue.cc:237] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.047909 21109 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:04.056161 21109 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.057171 21072 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.057695 21072 ts_tablet_manager.cc:1367] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.181s user 0.010s sys 0.011s
I20240506 15:14:04.058440 21073 tablet_bootstrap.cc:654] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.062495 21072 raft_consensus.cc:348] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.063344 21072 raft_consensus.cc:374] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.063657 21072 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.067302 21072 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.088913 21074 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.089645 21074 ts_tablet_manager.cc:1367] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.103s user 0.013s sys 0.003s
I20240506 15:14:04.093485 21074 raft_consensus.cc:348] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.094568 21074 raft_consensus.cc:374] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.094947 21074 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.108350 21074 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.118597 21073 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.119033 21073 ts_tablet_manager.cc:1367] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.004s
I20240506 15:14:04.121543 21072 ts_tablet_manager.cc:1398] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.064s user 0.012s sys 0.014s
I20240506 15:14:04.121239 21073 raft_consensus.cc:348] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.122005 21073 raft_consensus.cc:374] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.122303 21073 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.122584 21072 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.123000 21073 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.123950 21074 ts_tablet_manager.cc:1398] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
I20240506 15:14:04.124996 21074 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.125627 21073 ts_tablet_manager.cc:1398] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240506 15:14:04.126953 21073 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.129770 21072 tablet_bootstrap.cc:654] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.131392 21074 tablet_bootstrap.cc:654] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.130466 20780 catalog_manager.cc:5484] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.134337 21073 tablet_bootstrap.cc:654] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.142989 21074 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.143493 21074 ts_tablet_manager.cc:1367] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20240506 15:14:04.146237 21074 raft_consensus.cc:348] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.146976 21074 raft_consensus.cc:374] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.147264 21074 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.147976 21074 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.148312 20780 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.155345 21074 ts_tablet_manager.cc:1398] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240506 15:14:04.156441 21074 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.161761 21073 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.162295 21073 ts_tablet_manager.cc:1367] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.012s
I20240506 15:14:04.163537 21072 tablet_bootstrap.cc:492] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.163847 21074 tablet_bootstrap.cc:654] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.164183 21072 ts_tablet_manager.cc:1367] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.021s
I20240506 15:14:04.165153 21073 raft_consensus.cc:348] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.165967 21073 raft_consensus.cc:374] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.166280 21073 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.167153 21073 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.167670 21072 raft_consensus.cc:348] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.169740 21073 ts_tablet_manager.cc:1398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240506 15:14:04.171481 21073 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.173169 21072 raft_consensus.cc:374] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.173385 21074 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.173504 21072 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.174201 21074 ts_tablet_manager.cc:1367] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.017s
I20240506 15:14:04.174487 21072 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.176947 21072 ts_tablet_manager.cc:1398] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240506 15:14:04.177752 21073 tablet_bootstrap.cc:654] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.179220 21074 raft_consensus.cc:348] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.180006 21074 raft_consensus.cc:374] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.180354 21074 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.181089 21074 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.183161 21074 ts_tablet_manager.cc:1398] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20240506 15:14:04.184036 21074 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.190078 21074 tablet_bootstrap.cc:654] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.200238 21072 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.203336 21074 tablet_bootstrap.cc:492] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.203886 21074 ts_tablet_manager.cc:1367] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240506 15:14:04.206386 21073 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.206516 21074 raft_consensus.cc:348] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.207052 21072 tablet_bootstrap.cc:654] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.207089 21073 ts_tablet_manager.cc:1367] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.028s
I20240506 15:14:04.207360 21074 raft_consensus.cc:374] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.207976 21074 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.207060 20777 catalog_manager.cc:5484] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.208918 21074 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.210944 21073 raft_consensus.cc:348] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.212147 21073 raft_consensus.cc:374] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.212579 21073 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.213552 21073 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.216746 21073 ts_tablet_manager.cc:1398] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240506 15:14:04.218302 21073 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.218746 21074 ts_tablet_manager.cc:1398] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240506 15:14:04.223312 21074 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.223312 21107 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.223982 21107 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.226115 21107 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.227254 21073 tablet_bootstrap.cc:654] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.229137 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.229683 21074 tablet_bootstrap.cc:654] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.230404 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:04.232055 20921 leader_election.cc:341] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:04.233505 20924 leader_election.cc:341] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.234134 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.235842 21107 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.241751 21073 tablet_bootstrap.cc:492] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.242254 21073 ts_tablet_manager.cc:1367] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.003s
I20240506 15:14:04.243450 21112 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.243918 21112 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.244818 21073 raft_consensus.cc:348] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.245536 21073 raft_consensus.cc:374] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.245817 21073 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.246060 21112 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.246589 21073 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.247722 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:04.249115 20998 leader_election.cc:341] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.249212 21073 ts_tablet_manager.cc:1398] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240506 15:14:04.249931 21074 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.250341 21074 ts_tablet_manager.cc:1367] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.020s
I20240506 15:14:04.250407 21073 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.250555 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:04.251940 20996 leader_election.cc:341] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.252379 20996 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:04.252869 21074 raft_consensus.cc:348] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.253520 21112 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.253602 21074 raft_consensus.cc:374] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.254099 21074 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.254774 21074 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.256450 21073 tablet_bootstrap.cc:654] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.256625 21072 tablet_bootstrap.cc:492] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.256784 21074 ts_tablet_manager.cc:1398] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:14:04.257093 21072 ts_tablet_manager.cc:1367] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.057s user 0.023s sys 0.018s
I20240506 15:14:04.257884 21074 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.259756 21079 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.260221 21079 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.260421 21072 raft_consensus.cc:348] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.261499 21072 raft_consensus.cc:374] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.261898 21072 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.262400 21079 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.263865 21074 tablet_bootstrap.cc:654] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.263017 21072 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.264142 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.264832 20958 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:04.266396 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:04.269498 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.270102 21072 ts_tablet_manager.cc:1398] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20240506 15:14:04.271545 21072 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.275388 21074 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.275882 21074 ts_tablet_manager.cc:1367] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
W20240506 15:14:04.276384 20845 leader_election.cc:341] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.278342 21074 raft_consensus.cc:348] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.279080 21074 raft_consensus.cc:374] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.279227 21079 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.279371 21074 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.279906 21079 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.280352 21079 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.280462 21074 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.283416 21073 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.285811 21073 ts_tablet_manager.cc:1367] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.017s
I20240506 15:14:04.287846 21073 raft_consensus.cc:348] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.288591 21073 raft_consensus.cc:374] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.288854 21073 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.288913 20883 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.289574 21073 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.291637 21073 ts_tablet_manager.cc:1398] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 15:14:04.284624 21072 tablet_bootstrap.cc:654] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.292585 21073 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.293994 21079 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.297514 21074 ts_tablet_manager.cc:1398] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20240506 15:14:04.297530 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.298698 21074 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.298585 20958 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.300688 21112 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.301234 21112 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.303362 21112 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.305122 21074 tablet_bootstrap.cc:654] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.305646 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.306535 20882 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:04.307978 21112 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.308545 21112 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.309795 21073 tablet_bootstrap.cc:654] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.311654 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.312602 20882 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:04.314185 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:04.315860 20998 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.317073 21079 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.317358 21119 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.317963 21119 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.318588 21119 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:04.315979 20845 leader_election.cc:341] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.325939 21112 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.331995 21119 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.339709 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:04.340778 20882 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.334823 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.335784 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.342942 20956 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:04.346769 20996 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:04.348857 21112 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.352667 20882 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:04.354593 20998 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.355890 21077 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.358044 21119 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.359311 20958 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:04.359331 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.360245 20957 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.360757 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.361621 21079 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.362143 21079 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:04.370620 21079 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.372206 21140 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.373466 21140 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.376937 20957 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:04.377045 21077 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:04.378731 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.378371 21077 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:04.380502 20846 leader_election.cc:341] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.381225 21072 tablet_bootstrap.cc:492] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.381179 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.381748 21073 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.381979 21072 ts_tablet_manager.cc:1367] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.111s user 0.027s sys 0.044s
W20240506 15:14:04.382870 20845 leader_election.cc:341] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.382424 21140 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.383423 20845 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:04.383658 21140 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.385452 21074 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.385763 21073 ts_tablet_manager.cc:1367] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.014s
I20240506 15:14:04.386041 21074 ts_tablet_manager.cc:1367] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.088s user 0.036s sys 0.019s
I20240506 15:14:04.385577 21072 raft_consensus.cc:348] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.386528 21072 raft_consensus.cc:374] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.386852 21072 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.385385 21079 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.387399 21112 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.387632 21072 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.385530 21140 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.388325 21074 raft_consensus.cc:348] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.389256 21074 raft_consensus.cc:374] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.388970 21073 raft_consensus.cc:348] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.389678 21074 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.389724 21073 raft_consensus.cc:374] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.390091 21072 ts_tablet_manager.cc:1398] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240506 15:14:04.390388 21073 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.391090 21072 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.390609 21074 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.392877 21074 ts_tablet_manager.cc:1398] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240506 15:14:04.393803 21074 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.397038 21072 tablet_bootstrap.cc:654] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.397801 21140 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.399037 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.400978 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.401640 21031 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:04.403280 20845 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.404215 21140 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.404691 21140 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.405117 21140 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.402172 21073 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.408185 20957 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:04.410866 21112 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.411689 21112 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.413360 21073 ts_tablet_manager.cc:1398] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20240506 15:14:04.417351 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.419276 20883 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:04.420761 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.421917 21073 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.422542 21112 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.423275 21112 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.423704 21112 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.425506 21074 tablet_bootstrap.cc:654] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.424150 21112 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:04.431303 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:04.429075 20777 catalog_manager.cc:5484] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:04.440057 21140 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.444156 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.444744 21140 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.445096 20958 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.445483 21140 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.446483 20883 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.436976 21073 tablet_bootstrap.cc:654] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.441254 21112 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.448401 21107 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.446219 21140 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.449368 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:04.450007 20882 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.452483 21032 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.453728 21140 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.454016 20958 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:04.456653 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:04.457300 21031 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.456943 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.457669 21072 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.458163 21072 ts_tablet_manager.cc:1367] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.036s
W20240506 15:14:04.459419 20845 leader_election.cc:341] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.461148 20846 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:04.461601 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.462275 21140 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.462754 21140 raft_consensus.cc:686] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:04.462589 21072 raft_consensus.cc:348] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.463361 21072 raft_consensus.cc:374] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.463629 21140 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.463646 21072 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.448907 21107 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.465004 21072 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.466197 21107 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.462296 20956 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:04.466710 21031 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:04.467167 21072 ts_tablet_manager.cc:1398] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240506 15:14:04.467485 20882 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:04.468040 21072 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.467442 20846 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.468569 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.469305 21077 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.469795 21077 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:04.469326 21134 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.470739 21134 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.471097 21134 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.471006 21077 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.472501 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.473461 20882 raft_consensus.cc:2379] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:04.475399 21112 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.479394 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.480535 20956 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.481901 21073 tablet_bootstrap.cc:492] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.482429 21073 ts_tablet_manager.cc:1367] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.011s
I20240506 15:14:04.483839 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.491994 21107 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:04.494693 20921 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.495723 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.490976 20956 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:04.500149 21072 tablet_bootstrap.cc:654] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.499303 20780 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.492069 21073 raft_consensus.cc:348] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.501904 21073 raft_consensus.cc:374] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.502298 21073 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.500989 21107 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.503508 21073 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.504629 21109 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:04.504946 21080 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:04.505841 21134 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.510124 21074 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.510751 21074 ts_tablet_manager.cc:1367] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.117s user 0.028s sys 0.013s
I20240506 15:14:04.515656 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.516443 20957 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.517115 21134 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.521409 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:04.524521 20845 leader_election.cc:341] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.529078 21107 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:04.532824 20957 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:04.533337 21074 raft_consensus.cc:348] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.534081 21074 raft_consensus.cc:374] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.534343 21074 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.534835 21074 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.536206 20846 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.544847 21073 ts_tablet_manager.cc:1398] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.062s user 0.006s sys 0.001s
I20240506 15:14:04.546247 21073 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.565100 21134 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.565636 21134 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:04.577323 21074 ts_tablet_manager.cc:1398] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20240506 15:14:04.580216 21074 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.582577 21134 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.600507 21107 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:04.617957 21074 tablet_bootstrap.cc:654] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.622036 21072 tablet_bootstrap.cc:492] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.622725 21072 ts_tablet_manager.cc:1367] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.155s user 0.023s sys 0.007s
I20240506 15:14:04.632622 21073 tablet_bootstrap.cc:654] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.645030 21072 raft_consensus.cc:348] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.645841 21072 raft_consensus.cc:374] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.646175 21072 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.645689 20777 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:04.649514 21072 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.659741 21072 ts_tablet_manager.cc:1398] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20240506 15:14:04.671202 21080 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240506 15:14:04.681531 20778 catalog_manager.cc:5484] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.691278 21072 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.691398 21074 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.692013 21074 ts_tablet_manager.cc:1367] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.020s
I20240506 15:14:04.695096 21074 raft_consensus.cc:348] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.695844 21074 raft_consensus.cc:374] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.697276 21077 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.697757 21077 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.699816 21077 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.701069 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.696822 21074 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.701927 20957 raft_consensus.cc:2379] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:04.702086 21074 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.702378 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.704476 21074 ts_tablet_manager.cc:1398] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20240506 15:14:04.704236 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:04.705075 21073 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.705463 21074 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.705564 21073 ts_tablet_manager.cc:1367] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.160s user 0.013s sys 0.008s
I20240506 15:14:04.705744 21077 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.708192 21073 raft_consensus.cc:348] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.710024 21073 raft_consensus.cc:374] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.710422 21073 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.711318 21073 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.712726 21074 tablet_bootstrap.cc:654] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.713742 21073 ts_tablet_manager.cc:1398] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240506 15:14:04.714846 21073 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.716936 21077 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:04.720480 20778 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.723201 21073 tablet_bootstrap.cc:654] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.723948 21074 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.724633 21074 ts_tablet_manager.cc:1367] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20240506 15:14:04.727646 21074 raft_consensus.cc:348] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.728660 21074 raft_consensus.cc:374] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.729013 21074 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.730640 21074 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.730330 21079 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.732415 21079 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.734118 21074 ts_tablet_manager.cc:1398] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240506 15:14:04.734441 21079 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.736503 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.736503 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.740342 20957 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:04.742097 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:04.743340 21079 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.743847 21079 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:04.745311 20845 leader_election.cc:341] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.746603 21079 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.735380 21074 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.771131 21073 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.771792 21073 ts_tablet_manager.cc:1367] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.057s user 0.033s sys 0.021s
I20240506 15:14:04.773399 21072 tablet_bootstrap.cc:654] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.774335 21074 tablet_bootstrap.cc:654] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:04.777966 20845 consensus_peers.cc:485] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:04.777704 21079 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.774761 21073 raft_consensus.cc:348] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.779786 21079 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.780758 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.781646 20957 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.779263 21073 raft_consensus.cc:374] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.783779 21073 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.784533 21073 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.786671 21073 ts_tablet_manager.cc:1398] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240506 15:14:04.787590 21073 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.790078 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:04.791059 21074 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.791628 21074 ts_tablet_manager.cc:1367] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.011s
W20240506 15:14:04.792582 20845 leader_election.cc:341] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.793082 21073 tablet_bootstrap.cc:654] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.793732 20957 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:04.794080 21074 raft_consensus.cc:348] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.794835 21074 raft_consensus.cc:374] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.795125 21074 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.794950 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.795922 21074 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.796555 21079 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.797232 21079 raft_consensus.cc:686] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:04.798200 21074 ts_tablet_manager.cc:1398] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240506 15:14:04.799007 21074 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:04.798511 21079 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.803293 21072 tablet_bootstrap.cc:492] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.803813 21072 ts_tablet_manager.cc:1367] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.113s user 0.013s sys 0.000s
I20240506 15:14:04.805250 21074 tablet_bootstrap.cc:654] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.806282 21072 raft_consensus.cc:348] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.807060 21072 raft_consensus.cc:374] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.807317 21072 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.808910 21113 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.809623 21113 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.813352 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.814229 20881 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:04.815965 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:04.817804 21156 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.818290 21156 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.818694 21156 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.824584 21072 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.827685 21072 ts_tablet_manager.cc:1398] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240506 15:14:04.830698 21113 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.832158 21072 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.832335 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:04.834475 20921 leader_election.cc:341] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.836751 21139 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.837499 21139 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.845836 21072 tablet_bootstrap.cc:654] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.845784 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.846588 20881 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:04.848788 20998 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:04.844763 21139 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:04.850045 21112 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.850829 21112 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.851317 21112 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.855752 21073 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:04.856235 21073 ts_tablet_manager.cc:1367] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.069s user 0.030s sys 0.034s
I20240506 15:14:04.858677 21073 raft_consensus.cc:348] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.859350 21073 raft_consensus.cc:374] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.859617 21073 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.860217 21073 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.863783 21072 tablet_bootstrap.cc:492] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:04.860239 21077 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240506 15:14:04.864148 21072 ts_tablet_manager.cc:1367] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.000s
I20240506 15:14:04.870069 21112 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.871031 21073 ts_tablet_manager.cc:1398] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.014s user 0.001s sys 0.004s
I20240506 15:14:04.870920 21079 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:04.871997 21073 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:04.861275 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:04.872996 21072 raft_consensus.cc:348] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.865381 21139 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240506 15:14:04.873749 21072 raft_consensus.cc:374] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.874049 21072 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.869194 20780 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:04.864029 21156 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.881254 21072 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.887319 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:04.889483 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:04.889729 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:04.890260 20883 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.890727 21112 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:04.890988 20921 leader_election.cc:341] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.888108 21156 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.891631 21072 ts_tablet_manager.cc:1398] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240506 15:14:04.891607 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:04.892383 20882 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.880666 20957 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:04.900491 21073 tablet_bootstrap.cc:654] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.903733 21074 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:04.904244 21074 ts_tablet_manager.cc:1367] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.105s user 0.023s sys 0.019s
I20240506 15:14:04.904400 21072 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:04.907348 21074 raft_consensus.cc:348] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.908608 21074 raft_consensus.cc:374] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:04.909081 21074 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:04.930884 20883 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:04.932585 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.933877 21156 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.934434 21156 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:04.935256 21156 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.940734 21113 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.941479 21113 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.935513 20882 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:04.944629 20998 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:04.946163 21139 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:04.946784 21139 raft_consensus.cc:686] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:04.947757 21139 consensus_queue.cc:237] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.962491 21074 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:04.965494 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:04.966141 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:04.966822 20882 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
W20240506 15:14:04.967038 20921 leader_election.cc:341] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:04.967621 21113 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:04.968308 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:04.969806 21113 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:04.970178 21113 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:04.970549 21113 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:04.979094 21134 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:04.993654 21072 tablet_bootstrap.cc:654] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:04.999078 21077 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:04.999640 21077 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.002091 21077 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.010573 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.011680 20882 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.015879 21077 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.020857 21134 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.023887 21073 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.024452 21073 ts_tablet_manager.cc:1367] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.153s user 0.025s sys 0.042s
I20240506 15:14:05.027454 21073 raft_consensus.cc:348] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.028417 21073 raft_consensus.cc:374] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.028723 21073 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.029621 21073 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.032800 21073 ts_tablet_manager.cc:1398] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240506 15:14:05.034193 21073 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.036597 21113 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.041260 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:05.041379 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.042064 20883 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:05.042776 20921 leader_election.cc:341] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.042802 21134 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 15:14:05.043788 20998 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:05.045017 21112 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.045387 21112 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.045722 21112 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.051633 21134 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.052227 21134 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.052834 20883 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:05.054286 21134 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.055136 21073 tablet_bootstrap.cc:654] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.056006 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:05.058033 20845 leader_election.cc:341] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.060752 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.081194 21074 ts_tablet_manager.cc:1398] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.177s user 0.009s sys 0.010s
I20240506 15:14:05.082598 21074 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.069051 21113 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.079113 21107 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.094959 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:05.104455 20846 leader_election.cc:341] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.105197 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.106264 21134 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.107252 21113 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.108568 21113 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:05.110422 21113 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.100349 21107 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.117110 21107 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.121155 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:05.122921 20921 leader_election.cc:341] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.160494 21139 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.159970 20777 catalog_manager.cc:5484] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.161006 21139 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.162344 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.163013 20957 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.164666 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.147847 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.173727 21139 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:05.174736 20924 leader_election.cc:341] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.175235 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:05.177193 21156 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.179145 21140 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.198149 21176 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20240506 15:14:05.198801 21112 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.210744 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.215147 20881 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.216404 21112 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.243414 21073 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.244112 21073 ts_tablet_manager.cc:1367] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.210s user 0.036s sys 0.049s
I20240506 15:14:05.245518 21074 tablet_bootstrap.cc:654] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.246892 21073 raft_consensus.cc:348] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.247792 21073 raft_consensus.cc:374] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.248071 21073 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.248342 20881 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.249001 21073 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.249639 20998 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:05.250640 21184 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.251194 21184 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:05.251366 21073 ts_tablet_manager.cc:1398] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240506 15:14:05.252916 21073 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.252905 21184 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.230947 20778 catalog_manager.cc:5484] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:05.265074 21072 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.265571 21072 ts_tablet_manager.cc:1367] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.361s user 0.081s sys 0.074s
I20240506 15:14:05.266309 21073 tablet_bootstrap.cc:654] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.268090 21072 raft_consensus.cc:348] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.270697 21072 raft_consensus.cc:374] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.270975 21072 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.272305 21072 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.274392 21072 ts_tablet_manager.cc:1398] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20240506 15:14:05.274744 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 105985us
I20240506 15:14:05.275377 21072 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.279119 21074 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.280161 21074 ts_tablet_manager.cc:1367] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.198s user 0.010s sys 0.001s
I20240506 15:14:05.280774 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.284077 21074 raft_consensus.cc:348] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.285166 21074 raft_consensus.cc:374] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.285532 21074 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.286370 21074 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.289419 21074 ts_tablet_manager.cc:1398] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240506 15:14:05.291551 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.292603 20958 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.302937 21074 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.303905 21072 tablet_bootstrap.cc:654] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.304126 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.305843 21185 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.311918 20778 catalog_manager.cc:5484] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:05.313638 21073 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.314360 21073 ts_tablet_manager.cc:1367] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.002s
I20240506 15:14:05.318127 20958 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.303090 20779 catalog_manager.cc:5484] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.324715 21074 tablet_bootstrap.cc:654] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.329402 21073 raft_consensus.cc:348] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.330219 21073 raft_consensus.cc:374] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.330498 21073 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.331074 21072 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
W20240506 15:14:05.331463 20845 consensus_peers.cc:485] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:05.331688 21072 ts_tablet_manager.cc:1367] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.031s
I20240506 15:14:05.331894 21073 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.334105 21073 ts_tablet_manager.cc:1398] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20240506 15:14:05.334239 21072 raft_consensus.cc:348] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.334952 21072 raft_consensus.cc:374] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.335089 21073 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.335245 21072 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.335948 21072 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.337985 21072 ts_tablet_manager.cc:1398] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240506 15:14:05.338847 21072 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.345134 21072 tablet_bootstrap.cc:654] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.345736 20957 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.346011 21073 tablet_bootstrap.cc:654] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.357457 21185 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 15:14:05.361151 20921 consensus_peers.cc:485] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:05.365953 21074 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.366582 21074 ts_tablet_manager.cc:1367] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.064s user 0.019s sys 0.000s
I20240506 15:14:05.365957 21072 tablet_bootstrap.cc:492] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.368448 21072 ts_tablet_manager.cc:1367] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20240506 15:14:05.369464 21074 raft_consensus.cc:348] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.370424 21074 raft_consensus.cc:374] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.370935 21074 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.371301 21072 raft_consensus.cc:348] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.372813 21072 raft_consensus.cc:374] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.371582 21074 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.371958 21134 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.373291 21072 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.374174 21072 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.376233 21073 tablet_bootstrap.cc:492] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.376530 21072 ts_tablet_manager.cc:1398] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240506 15:14:05.376781 21073 ts_tablet_manager.cc:1367] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.003s
I20240506 15:14:05.377646 21072 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.386428 21184 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.388506 21032 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.389070 21072 tablet_bootstrap.cc:654] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.401551 21139 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.402101 21139 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.403204 21074 ts_tablet_manager.cc:1398] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20240506 15:14:05.404124 21139 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.404616 21074 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.405269 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.406006 20958 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.408075 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.407977 21073 raft_consensus.cc:348] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.408944 20881 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.409188 21073 raft_consensus.cc:374] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.409641 21073 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.410687 20996 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:05.411427 21073 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.412418 21139 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.412346 21184 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.412995 21184 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.415081 21184 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.415947 21139 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.416718 21139 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.417459 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:05.419116 20996 leader_election.cc:341] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.416067 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.420363 20881 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.422281 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.423477 21073 ts_tablet_manager.cc:1398] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.046s user 0.008s sys 0.000s
I20240506 15:14:05.423957 21184 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.435159 21184 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.435562 21184 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.426410 21185 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.441653 21113 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.441840 21185 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.445298 21134 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.445868 21134 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.446655 21185 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.447439 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.447885 21134 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.448136 20883 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.449573 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.450325 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:05.450575 20958 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:05.451094 21031 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:05.453076 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:05.453938 21134 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.454293 21134 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.454623 21134 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.454535 21184 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.454648 20998 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:05.456310 21185 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.460671 21073 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.461503 21139 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.463750 21074 tablet_bootstrap.cc:654] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.457322 21185 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.467522 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.468473 20958 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.469519 21185 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.469399 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.470085 20882 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.464469 21139 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.469439 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.477742 20883 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.481683 21184 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.484454 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:05.486510 20996 leader_election.cc:341] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.476358 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.492645 20882 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.494020 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.494733 21185 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.487458 20957 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.496836 21072 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
W20240506 15:14:05.496719 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:05.502718 21072 ts_tablet_manager.cc:1367] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.125s user 0.026s sys 0.053s
I20240506 15:14:05.505484 21072 raft_consensus.cc:348] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.505945 20958 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.506246 21072 raft_consensus.cc:374] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.498680 20883 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.506539 21072 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.507071 20996 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:05.507189 21072 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.507999 21139 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.509034 21139 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:05.502825 21184 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.509516 21072 ts_tablet_manager.cc:1398] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240506 15:14:05.510174 21184 raft_consensus.cc:686] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:05.509914 21139 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.510339 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.510985 20957 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.511150 21184 consensus_queue.cc:237] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.509222 21185 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.510459 21072 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.521457 20957 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.514478 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.496346 21134 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.525295 20996 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:05.526474 21077 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.527021 21077 raft_consensus.cc:686] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:05.528044 21077 consensus_queue.cc:237] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:05.531034 20921 consensus_peers.cc:485] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:05.531612 21073 tablet_bootstrap.cc:654] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.545076 21134 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.542573 20883 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:05.536407 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.550908 20957 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:05.547783 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:05.557677 20845 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.560160 21134 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.567467 21072 tablet_bootstrap.cc:654] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.572645 21074 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.573136 21074 ts_tablet_manager.cc:1367] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.169s user 0.019s sys 0.014s
I20240506 15:14:05.576207 21074 raft_consensus.cc:348] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.577317 21074 raft_consensus.cc:374] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.577710 21074 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.579066 21074 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.583518 21074 ts_tablet_manager.cc:1398] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20240506 15:14:05.584353 21072 tablet_bootstrap.cc:492] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.584479 21074 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.584913 21072 ts_tablet_manager.cc:1367] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.008s
I20240506 15:14:05.587409 21073 tablet_bootstrap.cc:492] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.588001 21073 ts_tablet_manager.cc:1367] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.006s
I20240506 15:14:05.587431 21072 raft_consensus.cc:348] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.588351 21072 raft_consensus.cc:374] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.588649 21072 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.589393 21072 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.591686 21074 tablet_bootstrap.cc:654] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.592526 21072 ts_tablet_manager.cc:1398] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240506 15:14:05.593389 21072 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.593232 21073 raft_consensus.cc:348] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.594449 21073 raft_consensus.cc:374] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.594709 21073 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.595304 21073 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.599007 21072 tablet_bootstrap.cc:654] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.605896 21073 ts_tablet_manager.cc:1398] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240506 15:14:05.607717 21073 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.611897 21072 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.609972 20778 catalog_manager.cc:5484] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.612483 21072 ts_tablet_manager.cc:1367] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20240506 15:14:05.613793 21073 tablet_bootstrap.cc:654] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.613641 20778 catalog_manager.cc:5484] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.615126 21072 raft_consensus.cc:348] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.616024 21072 raft_consensus.cc:374] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.616335 21072 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.616410 20778 catalog_manager.cc:5484] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.617079 21072 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.618767 21074 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.619467 21074 ts_tablet_manager.cc:1367] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.020s
I20240506 15:14:05.620608 21072 ts_tablet_manager.cc:1398] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20240506 15:14:05.621595 21072 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.621834 21074 raft_consensus.cc:348] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.622545 21074 raft_consensus.cc:374] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.622822 21074 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.623473 21074 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.626233 21074 ts_tablet_manager.cc:1398] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:14:05.628760 21074 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.629087 21072 tablet_bootstrap.cc:654] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.633709 21073 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.634248 21073 ts_tablet_manager.cc:1367] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20240506 15:14:05.635324 21074 tablet_bootstrap.cc:654] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.637005 21073 raft_consensus.cc:348] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.637890 21073 raft_consensus.cc:374] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.638175 21073 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.638854 21073 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.643002 21073 ts_tablet_manager.cc:1398] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240506 15:14:05.643923 21073 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.651639 21073 tablet_bootstrap.cc:654] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.658633 21134 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.659390 21134 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.663380 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.666551 21072 tablet_bootstrap.cc:492] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.667119 21072 ts_tablet_manager.cc:1367] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.012s
W20240506 15:14:05.669000 20846 leader_election.cc:341] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.669281 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:05.670629 20845 leader_election.cc:341] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.671027 21074 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.671065 20845 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:05.671650 21074 ts_tablet_manager.cc:1367] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.022s
I20240506 15:14:05.672124 21140 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.670935 21072 raft_consensus.cc:348] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.674046 21072 raft_consensus.cc:374] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.674341 21072 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.674974 21072 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.674918 21074 raft_consensus.cc:348] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.676651 21074 raft_consensus.cc:374] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.677362 21072 ts_tablet_manager.cc:1398] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20240506 15:14:05.678251 21073 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.678274 21072 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.678740 21073 ts_tablet_manager.cc:1367] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.013s
I20240506 15:14:05.681771 21073 raft_consensus.cc:348] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.682530 21073 raft_consensus.cc:374] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.682786 21073 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.676961 21074 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.683321 21134 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.684429 21072 tablet_bootstrap.cc:654] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.683871 21074 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.686429 21074 ts_tablet_manager.cc:1398] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240506 15:14:05.687311 21074 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.693212 21074 tablet_bootstrap.cc:654] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.693985 21080 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.694502 21080 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.698171 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.683367 21073 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:05.700176 20924 leader_election.cc:341] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.701985 21073 ts_tablet_manager.cc:1398] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20240506 15:14:05.703197 21073 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.710623 21073 tablet_bootstrap.cc:654] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.713388 21134 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.713932 21134 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.717466 21134 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.718523 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:05.720611 20845 leader_election.cc:341] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.724350 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.725071 20957 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:05.724385 21107 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 15:14:05.728166 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:05.731859 21134 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.732686 21134 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.733207 21134 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.736511 21080 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.765437 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:05.767774 21077 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:05.774518 21077 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.775136 21077 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:05.776208 20921 leader_election.cc:341] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.776782 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:05.778012 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.778939 21080 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:05.779579 21077 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.778865 20882 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.781471 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:05.781536 20998 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:05.782375 20958 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.782578 21134 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.784647 21134 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.785375 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.786188 20958 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.784025 21077 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.788491 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:05.789397 21073 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.789773 21073 ts_tablet_manager.cc:1367] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.005s
W20240506 15:14:05.790910 20845 leader_election.cc:341] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.787966 21077 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.795416 21077 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.795506 21031 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.801398 21184 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:05.804288 21077 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.807739 21077 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.792140 21073 raft_consensus.cc:348] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.808820 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:05.809345 21073 raft_consensus.cc:374] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.809559 20955 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.809947 21073 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.810803 21073 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.812940 21073 ts_tablet_manager.cc:1398] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.023s user 0.007s sys 0.001s
I20240506 15:14:05.813834 21073 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.815989 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:05.816676 20881 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.826012 20958 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:05.825933 20955 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.830340 21074 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.827980 20881 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:05.830796 21074 ts_tablet_manager.cc:1367] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.144s user 0.029s sys 0.037s
I20240506 15:14:05.830996 20996 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:05.831957 21077 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.832482 21077 raft_consensus.cc:686] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:05.833317 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:05.833287 21077 consensus_queue.cc:237] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.833966 21072 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.834409 21072 ts_tablet_manager.cc:1367] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.156s user 0.032s sys 0.023s
I20240506 15:14:05.834496 21140 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:05.834952 21140 raft_consensus.cc:686] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:05.835587 21140 consensus_queue.cc:237] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.837026 21072 raft_consensus.cc:348] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.838039 21072 raft_consensus.cc:374] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.838402 21072 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.839282 21072 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.833317 21074 raft_consensus.cc:348] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.841236 21074 raft_consensus.cc:374] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.841555 21074 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.842186 21074 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.842578 21072 ts_tablet_manager.cc:1398] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240506 15:14:05.843891 21072 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.845897 21031 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.852680 21073 tablet_bootstrap.cc:654] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.858003 21072 tablet_bootstrap.cc:654] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.865361 21073 tablet_bootstrap.cc:492] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.866020 21073 ts_tablet_manager.cc:1367] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.002s
I20240506 15:14:05.870205 21073 raft_consensus.cc:348] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.871212 21073 raft_consensus.cc:374] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.871616 21073 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.873445 21073 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.872699 20778 catalog_manager.cc:5484] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:05.880614 21074 ts_tablet_manager.cc:1398] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.049s user 0.020s sys 0.014s
I20240506 15:14:05.881593 21074 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.878470 21073 ts_tablet_manager.cc:1398] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20240506 15:14:05.883332 21073 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.894627 21073 tablet_bootstrap.cc:654] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.898365 21072 tablet_bootstrap.cc:492] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.898906 21072 ts_tablet_manager.cc:1367] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.010s
I20240506 15:14:05.901482 21072 raft_consensus.cc:348] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.902299 21072 raft_consensus.cc:374] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.902590 21074 tablet_bootstrap.cc:654] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.902748 21072 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.902083 20779 catalog_manager.cc:5484] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:05.903971 21072 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.906494 21072 ts_tablet_manager.cc:1398] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240506 15:14:05.907627 21072 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.912014 21074 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.912775 21074 ts_tablet_manager.cc:1367] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.000s
I20240506 15:14:05.917652 21072 tablet_bootstrap.cc:654] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.920704 21074 raft_consensus.cc:348] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.921586 21074 raft_consensus.cc:374] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.921914 21074 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.922605 21074 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.925137 21074 ts_tablet_manager.cc:1398] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240506 15:14:05.926137 21074 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:05.934919 21072 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:05.935725 21072 ts_tablet_manager.cc:1367] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.013s
I20240506 15:14:05.935263 21074 tablet_bootstrap.cc:654] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.937839 21072 raft_consensus.cc:348] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.940012 21072 raft_consensus.cc:374] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.940361 21072 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.941221 21072 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.943539 21077 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.943989 21077 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.944401 21072 ts_tablet_manager.cc:1398] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240506 15:14:05.945798 21072 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:05.949801 21184 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.952046 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:05.952828 20881 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:05.953446 21072 tablet_bootstrap.cc:654] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:05.952919 21031 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.959617 21073 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:05.959973 20998 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:05.960321 21073 ts_tablet_manager.cc:1367] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.026s
I20240506 15:14:05.960943 21184 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:05.961337 21184 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:05.961786 21184 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:05.964056 21077 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:05.965373 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:05.967541 20996 leader_election.cc:341] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:05.964064 21073 raft_consensus.cc:348] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.969321 21073 raft_consensus.cc:374] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:05.969805 21073 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:05.970686 21073 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.981364 21073 ts_tablet_manager.cc:1398] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.021s user 0.007s sys 0.002s
I20240506 15:14:05.987892 21185 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.987985 21077 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:05.990643 21073 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:05.990201 21156 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:05.994890 21156 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:05.996109 21074 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:05.996801 21074 ts_tablet_manager.cc:1367] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.071s user 0.035s sys 0.011s
I20240506 15:14:05.997191 21156 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:05.989929 21184 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.000882 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.001695 21032 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.004539 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:06.047534 21222 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.048434 21222 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.008441 21184 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:06.006963 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.028033 21156 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.054579 20880 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.056149 21156 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.058035 21156 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.055615 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:06.059077 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.027081 21079 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.059774 20958 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:06.060703 20879 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.060801 21079 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.025123 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.071084 21222 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.078788 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.081876 21072 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:06.082347 21072 ts_tablet_manager.cc:1367] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.137s user 0.017s sys 0.005s
I20240506 15:14:06.083065 21134 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.083465 21134 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.083806 21134 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.082836 21184 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:06.085193 21074 raft_consensus.cc:348] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.085858 21074 raft_consensus.cc:374] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.086110 21074 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.086699 21074 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.088308 21074 ts_tablet_manager.cc:1398] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.091s user 0.016s sys 0.027s
I20240506 15:14:06.089156 21074 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.089953 21072 raft_consensus.cc:348] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.090629 21072 raft_consensus.cc:374] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.090898 21072 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.091411 21073 tablet_bootstrap.cc:654] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.077733 21080 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.093452 21072 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.094671 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.095299 21031 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:06.097349 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:06.099951 20845 leader_election.cc:341] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.097352 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:06.100874 21074 tablet_bootstrap.cc:654] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.101135 20955 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
W20240506 15:14:06.076390 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 68448.1us
I20240506 15:14:06.086799 21079 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.104104 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.093276 21080 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.105386 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:06.106528 21080 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.113476 20957 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:06.128950 20996 leader_election.cc:341] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.136688 20921 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.138710 21112 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.140544 21134 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.127892 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.145700 21080 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.147104 21079 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.147475 21079 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.147827 21079 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.152827 21072 ts_tablet_manager.cc:1398] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.070s user 0.004s sys 0.008s
I20240506 15:14:06.153182 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:06.115283 20880 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:06.154202 21072 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:06.154892 20998 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:06.156576 21112 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.157047 21112 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:06.161218 20958 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:06.161960 21134 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.162739 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:06.163697 20880 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:06.163789 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.167166 21032 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.168675 21113 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:06.176057 21080 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.157845 21112 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.182269 21107 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.187078 21107 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.189103 21107 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.193329 21156 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.193835 21156 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.196951 21156 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.200316 21079 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.203868 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.204842 20880 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.211918 21235 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.212617 21235 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.219203 21074 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:06.225782 21032 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:06.234913 21072 tablet_bootstrap.cc:654] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.237135 21074 ts_tablet_manager.cc:1367] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.008s
I20240506 15:14:06.237493 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.241145 21073 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.241689 21031 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:06.242717 20924 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.243465 21074 raft_consensus.cc:348] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.244342 21074 raft_consensus.cc:374] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.244390 21107 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.244712 21074 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.244941 21107 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.245364 21107 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.245400 21074 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.245234 21079 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.250362 21072 tablet_bootstrap.cc:492] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:06.250981 21072 ts_tablet_manager.cc:1367] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.009s
I20240506 15:14:06.251762 21074 ts_tablet_manager.cc:1398] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240506 15:14:06.241652 21073 ts_tablet_manager.cc:1367] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.251s user 0.020s sys 0.013s
I20240506 15:14:06.252709 21074 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.255110 21072 raft_consensus.cc:348] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.255973 21072 raft_consensus.cc:374] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.256325 21072 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.255267 21073 raft_consensus.cc:348] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.257156 21073 raft_consensus.cc:374] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.228988 21235 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:06.258899 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:06.260309 21185 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:06.260982 20958 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.261215 21074 tablet_bootstrap.cc:654] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.258600 21073 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.252734 21107 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.257685 21072 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.264458 21107 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.238534 20845 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:06.265601 21079 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.266069 21079 raft_consensus.cc:686] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:06.266997 21079 consensus_queue.cc:237] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.269240 20958 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:06.271529 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.273000 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:06.273388 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.273854 20957 raft_consensus.cc:2379] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:06.274323 21134 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.274842 21134 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:06.275141 21156 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.275714 21156 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.276059 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.276670 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.277319 20880 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.277920 21156 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.278184 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:06.278326 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.278931 20883 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.279376 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.279922 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.280575 20880 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.280934 21107 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.281297 21107 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.281620 21107 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.282548 20924 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.283744 21113 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.284143 21113 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.284518 21113 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.275794 21134 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.288513 20883 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:06.292923 21072 ts_tablet_manager.cc:1398] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20240506 15:14:06.293967 21072 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:06.293697 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:06.294685 20845 leader_election.cc:341] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.295984 20924 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.297049 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.298787 21156 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240506 15:14:06.299901 20921 leader_election.cc:341] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.300014 21156 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
W20240506 15:14:06.301164 20921 leader_election.cc:341] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.300740 21156 raft_consensus.cc:686] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:06.301239 21113 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.303036 21156 consensus_queue.cc:237] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.303647 20777 catalog_manager.cc:5484] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:06.307201 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.308120 21032 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.263803 21073 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.309376 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:06.310338 21112 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.311079 21074 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:06.313288 21074 ts_tablet_manager.cc:1367] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.003s
I20240506 15:14:06.311256 21073 ts_tablet_manager.cc:1398] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.058s user 0.003s sys 0.001s
I20240506 15:14:06.316604 21074 raft_consensus.cc:348] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.317975 21113 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.319334 21074 raft_consensus.cc:374] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.322198 21112 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:06.319772 21074 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.323293 21074 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.324117 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:06.325068 20881 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.325568 21074 ts_tablet_manager.cc:1398] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240506 15:14:06.326428 21074 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.327478 21073 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.332909 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.335549 21032 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:06.382562 21073 tablet_bootstrap.cc:654] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.378295 21112 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240506 15:14:06.388970 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.389644 21032 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.375486 20881 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:06.393782 20924 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.394604 21113 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.399047 20777 catalog_manager.cc:5484] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:06.348430 21107 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.395364 21113 raft_consensus.cc:686] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:06.409739 21134 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.373798 20779 catalog_manager.cc:5484] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:06.422727 21113 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.423547 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:06.423547 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.424018 21072 tablet_bootstrap.cc:654] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.424361 20883 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.424419 21031 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.428897 21032 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:06.420142 21107 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.448693 21074 tablet_bootstrap.cc:654] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.450400 21031 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:06.468103 20921 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:06.469012 21107 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.469547 21107 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:06.470419 21107 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.465967 21113 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.474777 21235 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.478226 21134 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:06.480811 21113 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.484972 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.485810 20883 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:06.477540 20780 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:06.489517 20924 leader_election.cc:341] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.493377 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.494035 21113 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:06.499445 20921 leader_election.cc:341] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.510131 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.511217 21113 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:06.521757 21073 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.522459 21073 ts_tablet_manager.cc:1367] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.195s user 0.018s sys 0.012s
I20240506 15:14:06.531828 21072 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:06.532395 21072 ts_tablet_manager.cc:1367] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.239s user 0.023s sys 0.042s
I20240506 15:14:06.534351 20779 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:06.539211 21073 raft_consensus.cc:348] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.540016 21073 raft_consensus.cc:374] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.539749 21072 raft_consensus.cc:348] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.540364 21073 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.541095 21073 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.540670 21072 raft_consensus.cc:374] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.542240 21072 raft_consensus.cc:729] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.543635 21072 consensus_queue.cc:260] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.544962 21074 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:06.545550 21074 ts_tablet_manager.cc:1367] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.219s user 0.024s sys 0.019s
I20240506 15:14:06.546236 21073 ts_tablet_manager.cc:1398] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240506 15:14:06.547158 21073 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.548480 21072 ts_tablet_manager.cc:1398] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20240506 15:14:06.548652 21074 raft_consensus.cc:348] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.549580 21074 raft_consensus.cc:374] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.550187 21074 raft_consensus.cc:729] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.551523 21074 consensus_queue.cc:260] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.553141 21073 tablet_bootstrap.cc:654] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.555145 21074 ts_tablet_manager.cc:1398] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240506 15:14:06.559655 21074 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.549429 21072 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:06.566040 21073 tablet_bootstrap.cc:492] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.567785 21073 ts_tablet_manager.cc:1367] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20240506 15:14:06.567796 21074 tablet_bootstrap.cc:654] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.570498 21073 raft_consensus.cc:348] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.571271 21073 raft_consensus.cc:374] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.571578 21073 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.572582 21073 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.575011 21073 ts_tablet_manager.cc:1398] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240506 15:14:06.575969 21073 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.581102 21134 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.581640 21134 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.583293 21073 tablet_bootstrap.cc:654] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.583636 21134 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.585861 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.586658 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:06.587399 20845 leader_election.cc:341] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:06.588205 20846 leader_election.cc:341] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.588678 20846 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:06.590091 21134 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:06.593302 21074 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:06.593408 21113 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.571435 21072 tablet_bootstrap.cc:654] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.593886 21113 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.594338 20778 catalog_manager.cc:5484] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:06.598392 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:06.598601 21113 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.599314 20880 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:14:06.601136 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:06.601253 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.593837 21074 ts_tablet_manager.cc:1367] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.006s
I20240506 15:14:06.603619 20921 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.604801 21113 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.605018 21074 raft_consensus.cc:348] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.606014 21074 raft_consensus.cc:374] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.606367 21074 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.607317 21074 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.612211 21113 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:06.616969 21074 ts_tablet_manager.cc:1398] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.015s user 0.008s sys 0.001s
I20240506 15:14:06.619432 21074 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.620183 21073 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.620695 21073 ts_tablet_manager.cc:1367] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.015s
I20240506 15:14:06.623154 21073 raft_consensus.cc:348] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.623883 21073 raft_consensus.cc:374] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.624137 21073 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.624756 21073 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.626928 21073 ts_tablet_manager.cc:1398] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240506 15:14:06.627910 21073 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.630028 21074 tablet_bootstrap.cc:654] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.632946 21185 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.633569 21073 tablet_bootstrap.cc:654] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.649091 21072 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:06.649705 21072 ts_tablet_manager.cc:1367] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.013s
I20240506 15:14:06.652233 21235 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.654757 21072 raft_consensus.cc:348] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.655591 21072 raft_consensus.cc:374] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.655697 21073 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.655978 21072 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.656212 21073 ts_tablet_manager.cc:1367] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.001s
I20240506 15:14:06.656869 21072 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.659150 21072 ts_tablet_manager.cc:1398] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240506 15:14:06.662106 21072 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:06.659385 21073 raft_consensus.cc:348] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.663323 21073 raft_consensus.cc:374] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.663631 21073 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.664294 21073 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.665881 21073 ts_tablet_manager.cc:1398] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240506 15:14:06.666752 21073 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.672142 21073 tablet_bootstrap.cc:654] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.677233 21072 tablet_bootstrap.cc:654] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.680287 21140 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.680848 21140 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.682905 21140 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.684698 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:06.685765 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:06.686257 20845 leader_election.cc:341] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.686408 20958 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:06.687461 21074 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:06.686893 21140 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.687916 21074 ts_tablet_manager.cc:1367] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.000s
I20240506 15:14:06.688802 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.689563 21140 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:06.690168 21140 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:06.690574 21140 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:06.690469 21074 raft_consensus.cc:348] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.691210 21074 raft_consensus.cc:374] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.691529 21074 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.694988 21074 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.696173 21073 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:06.696661 21073 ts_tablet_manager.cc:1367] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.001s
I20240506 15:14:06.697552 21074 ts_tablet_manager.cc:1398] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240506 15:14:06.698844 21074 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:06.701267 21073 raft_consensus.cc:348] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.702050 21073 raft_consensus.cc:374] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:06.702342 21073 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:06.706238 21074 tablet_bootstrap.cc:654] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.704491 21140 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.710381 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:06.710659 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:06.711005 20954 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:06.712872 20845 leader_election.cc:341] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:06.716806 21222 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:06.722134 20954 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:06.724426 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:06.725302 21222 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:06.725878 21222 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:06.726963 21222 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.730523 21107 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.720985 21073 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.732779 21140 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.731207 21107 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.749867 21073 ts_tablet_manager.cc:1398] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20240506 15:14:06.750982 21073 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:06.816704 21222 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.817476 21222 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.969055 21073 tablet_bootstrap.cc:700] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Time spent opening tablet: real 0.214s user 0.004s sys 0.000s
I20240506 15:14:06.970021 21073 tablet_bootstrap.cc:654] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:06.970270 21222 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:06.970801 21140 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.974651 21113 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:06.981061 21072 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:06.981760 21072 ts_tablet_manager.cc:1367] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.320s user 0.012s sys 0.023s
I20240506 15:14:06.971688 21140 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:06.975227 21113 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.006017 21072 raft_consensus.cc:348] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.007099 21072 raft_consensus.cc:374] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.007562 21072 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.012948 21072 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.000769 21107 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.001560 21112 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.018682 21072 ts_tablet_manager.cc:1398] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20240506 15:14:07.020570 21072 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:07.036238 21074 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:07.037025 21074 ts_tablet_manager.cc:1367] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.338s user 0.022s sys 0.029s
I20240506 15:14:07.022259 21080 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.063894 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.066882 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.067018 21113 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.067615 20883 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:07.061121 21080 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.070183 21080 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.071166 21140 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.068017 21074 raft_consensus.cc:348] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.072759 21074 raft_consensus.cc:374] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.073093 21074 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.073628 21074 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.082232 21074 ts_tablet_manager.cc:1398] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20240506 15:14:07.025213 21112 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:07.056203 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 71973us
I20240506 15:14:07.084117 21074 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
W20240506 15:14:07.089706 20845 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.086594 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.091297 20954 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:07.102478 20778 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:07.151413 20845 consensus_peers.cc:485] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:07.124107 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.200757 21275 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.201992 21275 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.197034 21029 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:07.210525 21112 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:07.221736 21267 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.142s user 0.001s sys 0.001s
I20240506 15:14:07.223421 21073 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:07.223896 21073 ts_tablet_manager.cc:1367] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.473s user 0.014s sys 0.006s
I20240506 15:14:07.219341 21274 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.220826 21275 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:07.232820 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.234742 21072 tablet_bootstrap.cc:654] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.238494 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.242386 20878 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:07.243544 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.237413 21274 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.259033 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.253822 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.253144 21274 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.263193 21272 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.263254 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.263742 21272 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.264125 21272 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.263998 20924 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.266654 21073 raft_consensus.cc:348] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.267319 21073 raft_consensus.cc:374] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.263768 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.267596 21073 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.267668 21270 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:07.266920 21274 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.268784 21274 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.269114 21274 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.269038 21073 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.271378 21073 ts_tablet_manager.cc:1398] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20240506 15:14:07.271822 21074 tablet_bootstrap.cc:654] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.261384 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.272337 21073 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:07.277256 20845 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:07.278373 21079 raft_consensus.cc:2781] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.278878 21079 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.279346 21079 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:07.293705 20996 leader_election.cc:341] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:07.292753 20845 consensus_peers.cc:485] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:07.307794 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.311832 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:07.314334 20996 leader_election.cc:341] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:07.324998 20924 leader_election.cc:341] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.329077 21074 tablet_bootstrap.cc:492] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:07.329597 21074 ts_tablet_manager.cc:1367] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.246s user 0.014s sys 0.000s
I20240506 15:14:07.331220 21073 tablet_bootstrap.cc:654] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.332556 21222 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.334134 21222 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.337378 21222 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.337585 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.341866 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.343593 21074 raft_consensus.cc:348] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.344022 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.344380 21074 raft_consensus.cc:374] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.344674 21074 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.342687 21274 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.346323 21270 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 15:14:07.349498 20921 consensus_peers.cc:485] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:07.359360 21274 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.359089 21074 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.360821 21272 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.359119 21072 tablet_bootstrap.cc:492] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:07.362633 21072 ts_tablet_manager.cc:1367] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.342s user 0.020s sys 0.024s
I20240506 15:14:07.363314 21272 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:07.365368 20921 leader_election.cc:341] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:07.366273 20921 leader_election.cc:341] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.366717 20921 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.367313 21074 ts_tablet_manager.cc:1398] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.031s user 0.003s sys 0.005s
I20240506 15:14:07.368242 21074 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:07.371011 21156 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:07.371645 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.372579 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.373400 21270 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240506 15:14:07.373420 20998 leader_election.cc:341] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.373744 21156 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:07.374039 21270 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.374054 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.374720 21270 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.375460 21279 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.336817 21079 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.375458 21072 raft_consensus.cc:348] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.377264 21072 raft_consensus.cc:374] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.377560 21072 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.380615 21274 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.381052 20877 raft_consensus.cc:2379] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:07.382346 20846 leader_election.cc:341] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:07.376681 20845 leader_election.cc:341] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.384209 21079 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.385429 21079 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:07.385887 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.386234 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.386945 20879 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:07.378150 21072 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.388808 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:07.390482 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.390594 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:07.390851 20883 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.391234 20954 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:07.388774 21274 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240506 15:14:07.393632 21072 ts_tablet_manager.cc:1398] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
W20240506 15:14:07.394167 20921 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.394701 21072 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
W20240506 15:14:07.395897 20921 leader_election.cc:341] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.396924 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:07.403264 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.413061 21030 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:07.414584 20845 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.417335 21072 tablet_bootstrap.cc:654] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.425489 21273 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.425926 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.426048 21273 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.432512 21274 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.432969 21274 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.434985 21274 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.439983 21073 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:07.440569 21073 ts_tablet_manager.cc:1367] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.168s user 0.024s sys 0.014s
I20240506 15:14:07.445325 21273 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.446185 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.446034 21140 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:07.452360 20998 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.463074 21222 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.467056 21222 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.467521 21222 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.463037 20883 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
W20240506 15:14:07.477254 20846 leader_election.cc:341] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.494735 21073 raft_consensus.cc:348] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.496817 21073 raft_consensus.cc:374] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.497143 21073 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.497766 21073 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.483108 21030 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:07.503576 21281 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.504750 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:07.505704 21029 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:07.509025 21073 ts_tablet_manager.cc:1398] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20240506 15:14:07.509984 21073 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
W20240506 15:14:07.510655 20921 leader_election.cc:341] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.513924 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.515259 21273 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.515796 21273 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.516304 21273 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.522166 21222 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.524720 21222 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.524394 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.522902 20845 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.537537 21074 tablet_bootstrap.cc:654] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.538344 20883 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:07.543404 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.543879 21270 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.545152 20958 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:07.551252 20924 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:07.551470 20845 leader_election.cc:341] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.553102 21107 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:07.553647 21107 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:07.554445 21107 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.545475 21222 raft_consensus.cc:2781] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:07.568560 21072 tablet_bootstrap.cc:492] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:07.569206 21072 ts_tablet_manager.cc:1367] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.175s user 0.013s sys 0.009s
I20240506 15:14:07.574008 21235 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.551174 21270 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.576668 21235 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.568488 21222 raft_consensus.cc:686] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:07.559895 21274 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.588696 21072 raft_consensus.cc:348] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.590317 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.590570 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.615998 21272 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.591558 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:07.617303 20881 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:07.618085 20778 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:07.593909 21289 log.cc:924] Time spent T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20240506 15:14:07.595235 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.595513 21222 consensus_queue.cc:237] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:07.596539 21290 log.cc:924] Time spent T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240506 15:14:07.600651 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:07.591581 21235 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:07.602103 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.602777 21274 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.604396 21273 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.607223 21072 raft_consensus.cc:374] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.641228 21072 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.636571 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:07.640002 21073 tablet_bootstrap.cc:654] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.641983 21072 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.651244 20958 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:07.657369 21072 ts_tablet_manager.cc:1398] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.088s user 0.005s sys 0.004s
I20240506 15:14:07.661098 21074 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:07.661312 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.664192 21074 ts_tablet_manager.cc:1367] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.296s user 0.026s sys 0.021s
I20240506 15:14:07.667867 21074 raft_consensus.cc:348] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.668964 21074 raft_consensus.cc:374] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.669248 21074 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.669770 21074 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.673214 20877 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:07.681349 20921 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.680012 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.707406 21279 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.706008 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.709749 21274 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.713222 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:07.725627 21072 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:07.737316 21074 ts_tablet_manager.cc:1398] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.073s user 0.007s sys 0.011s
I20240506 15:14:07.738291 21308 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.756006 21074 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:07.730405 21273 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:07.739933 20921 leader_election.cc:341] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.743990 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:07.771561 21073 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:07.772279 21073 ts_tablet_manager.cc:1367] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.262s user 0.017s sys 0.012s
I20240506 15:14:07.775811 20877 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.774256 20921 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.798503 21029 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.800546 21301 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:07.806077 21314 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.806612 21314 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:07.804970 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 70034.4us
I20240506 15:14:07.814040 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.814726 21030 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.824898 21072 tablet_bootstrap.cc:654] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.852025 20778 catalog_manager.cc:5484] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:07.855316 21270 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:07.855856 21270 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.861635 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:07.862387 21270 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.862439 20958 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:07.863014 21314 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.828423 20877 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:07.868361 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:07.870662 21030 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:07.871289 20924 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.872437 21270 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:07.873070 21270 raft_consensus.cc:686] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:07.872964 21073 raft_consensus.cc:348] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.875532 21073 raft_consensus.cc:374] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.876650 21073 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.877624 21073 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.879633 21073 ts_tablet_manager.cc:1398] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.107s user 0.007s sys 0.000s
I20240506 15:14:07.880901 21073 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
W20240506 15:14:07.874795 20924 leader_election.cc:341] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.874351 21270 consensus_queue.cc:237] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.886226 21074 tablet_bootstrap.cc:654] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.888072 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.889063 21314 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:07.889511 21314 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:07.889897 21314 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.889341 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:07.904877 20845 leader_election.cc:341] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.872992 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:07.912984 20921 leader_election.cc:341] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:07.915850 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:07.915598 21314 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.928340 21107 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 15:14:07.924829 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:07.932577 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:07.933214 20957 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.941219 21072 tablet_bootstrap.cc:492] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:07.941751 21072 ts_tablet_manager.cc:1367] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.216s user 0.023s sys 0.019s
I20240506 15:14:07.942575 20957 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:07.945533 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:07.947104 21074 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:07.947646 21074 ts_tablet_manager.cc:1367] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.192s user 0.012s sys 0.006s
I20240506 15:14:07.947207 21311 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:07.948292 21311 raft_consensus.cc:686] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:07.944228 21072 raft_consensus.cc:348] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.949173 21311 consensus_queue.cc:237] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.950305 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:07.949187 21072 raft_consensus.cc:374] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.953202 21072 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.953953 21072 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.953810 20779 catalog_manager.cc:5484] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:07.955354 21029 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:07.962813 21029 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:07.944216 21314 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:07.952845 21073 tablet_bootstrap.cc:654] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.965690 21074 raft_consensus.cc:348] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.967461 21074 raft_consensus.cc:374] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:07.967793 21074 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:07.968550 21074 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:07.970674 21074 ts_tablet_manager.cc:1398] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.023s user 0.003s sys 0.005s
I20240506 15:14:07.970840 21072 ts_tablet_manager.cc:1398] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.029s user 0.001s sys 0.005s
I20240506 15:14:07.971648 21074 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:07.972728 21072 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:07.979138 21074 tablet_bootstrap.cc:654] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.981849 21072 tablet_bootstrap.cc:654] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:07.990432 20780 catalog_manager.cc:5484] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:07.994969 21270 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240506 15:14:08.003701 20921 consensus_peers.cc:485] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:08.008186 21073 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.009126 21073 ts_tablet_manager.cc:1367] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.000s
I20240506 15:14:08.012818 21073 raft_consensus.cc:348] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.013564 21073 raft_consensus.cc:374] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.013846 21073 raft_consensus.cc:729] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.018353 21073 consensus_queue.cc:260] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.026039 21072 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.026829 21072 ts_tablet_manager.cc:1367] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.026s
I20240506 15:14:08.029450 21072 raft_consensus.cc:348] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.030236 21072 raft_consensus.cc:374] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.030508 21072 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.031248 21072 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.034853 21073 ts_tablet_manager.cc:1398] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20240506 15:14:08.038220 21072 ts_tablet_manager.cc:1398] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20240506 15:14:08.039160 21072 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.043841 21073 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.051177 21072 tablet_bootstrap.cc:654] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.056308 21030 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.059877 21074 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.060576 21074 ts_tablet_manager.cc:1367] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.036s
I20240506 15:14:08.069149 21074 raft_consensus.cc:348] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.070013 21074 raft_consensus.cc:374] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.070377 21074 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.071019 21074 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.073172 21074 ts_tablet_manager.cc:1398] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20240506 15:14:08.074102 21074 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.076088 21073 tablet_bootstrap.cc:654] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.077415 21072 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.077894 21072 ts_tablet_manager.cc:1367] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.009s
I20240506 15:14:08.085176 21072 raft_consensus.cc:348] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.085966 21072 raft_consensus.cc:374] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.086248 21072 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.087356 21072 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.089578 21072 ts_tablet_manager.cc:1398] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240506 15:14:08.090526 21072 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.090839 21073 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.091450 21073 ts_tablet_manager.cc:1367] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.000s
I20240506 15:14:08.093526 21074 tablet_bootstrap.cc:654] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.094460 21073 raft_consensus.cc:348] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.097004 21073 raft_consensus.cc:374] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.097358 21073 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.098254 21073 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.101167 21073 ts_tablet_manager.cc:1398] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240506 15:14:08.101974 21073 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.103181 21074 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.103631 21074 ts_tablet_manager.cc:1367] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.002s
I20240506 15:14:08.096064 21072 tablet_bootstrap.cc:654] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.110615 21073 tablet_bootstrap.cc:654] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.106479 21074 raft_consensus.cc:348] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.111454 21074 raft_consensus.cc:374] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.111770 21074 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.120520 21074 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.124121 21074 ts_tablet_manager.cc:1398] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20240506 15:14:08.127859 21074 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.139914 21072 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.140487 21072 ts_tablet_manager.cc:1367] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.050s user 0.019s sys 0.002s
I20240506 15:14:08.143520 21074 tablet_bootstrap.cc:654] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.143661 21314 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:08.152748 21074 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.153517 21074 ts_tablet_manager.cc:1367] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.008s
I20240506 15:14:08.154479 21314 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.155510 21074 raft_consensus.cc:348] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.156394 21074 raft_consensus.cc:374] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.156706 21074 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.156525 21314 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.157420 21074 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.159799 21314 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.159855 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.160745 20957 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:08.162046 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.162899 21073 tablet_bootstrap.cc:492] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.163372 21073 ts_tablet_manager.cc:1367] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.025s
I20240506 15:14:08.163298 21314 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.163867 21314 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.164109 21074 ts_tablet_manager.cc:1398] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20240506 15:14:08.164449 21314 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.165096 21074 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.166071 21073 raft_consensus.cc:348] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.173170 21073 raft_consensus.cc:374] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.173491 21073 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.174098 21073 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.176203 21073 ts_tablet_manager.cc:1398] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20240506 15:14:08.177145 21073 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.179121 21074 tablet_bootstrap.cc:654] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.166846 21072 raft_consensus.cc:348] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.180418 21072 raft_consensus.cc:374] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.180730 21072 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.181370 21072 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.183495 21072 ts_tablet_manager.cc:1398] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20240506 15:14:08.184448 21072 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.194515 21314 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.198508 21073 tablet_bootstrap.cc:654] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.200958 21314 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.203895 21072 tablet_bootstrap.cc:654] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.209331 20958 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.217979 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.220814 21072 tablet_bootstrap.cc:492] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.221331 21072 ts_tablet_manager.cc:1367] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.003s
I20240506 15:14:08.227988 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:08.228399 21073 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.228729 20957 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.228863 21073 ts_tablet_manager.cc:1367] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.004s
I20240506 15:14:08.231658 21073 raft_consensus.cc:348] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.232367 21073 raft_consensus.cc:374] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.232651 21073 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.223745 21072 raft_consensus.cc:348] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.233453 21072 raft_consensus.cc:374] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.233754 21072 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
W20240506 15:14:08.233939 20845 leader_election.cc:341] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.234481 21072 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.235363 21073 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.242884 21073 ts_tablet_manager.cc:1398] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240506 15:14:08.244333 21073 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.244721 20957 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:08.240677 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.246126 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:08.247556 20845 leader_election.cc:341] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.248277 21079 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.249372 21074 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.249845 21074 ts_tablet_manager.cc:1367] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.085s user 0.045s sys 0.025s
I20240506 15:14:08.249698 21079 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:08.250741 21079 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.252339 21074 raft_consensus.cc:348] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.253046 21072 ts_tablet_manager.cc:1398] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20240506 15:14:08.253950 21072 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.253065 21074 raft_consensus.cc:374] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.261497 21074 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.261752 21277 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.262281 21277 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.262626 21074 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.264286 21314 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:08.264766 21074 ts_tablet_manager.cc:1398] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240506 15:14:08.265686 21074 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.265990 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.266757 20957 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:08.268014 21277 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.270043 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.270890 20877 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:08.272153 20998 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.273522 21235 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.274749 21272 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.275326 21272 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.276906 21072 tablet_bootstrap.cc:654] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.278595 21272 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.279181 21279 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.279939 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.279829 21279 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:08.282747 20845 leader_election.cc:341] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.284293 21279 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.285141 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.286952 20777 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:08.298187 21235 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.300843 21074 tablet_bootstrap.cc:654] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.301450 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.309250 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:08.311789 20846 leader_election.cc:341] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.312400 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.312969 21073 tablet_bootstrap.cc:654] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.313869 20998 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.317860 21079 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.320407 21272 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.320953 21272 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.322953 21272 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.316690 21235 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.326236 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:08.336671 20845 leader_election.cc:341] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.343027 21272 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.344130 21272 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.349503 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.350579 21029 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:08.353013 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.353741 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:08.354972 21222 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.355499 21222 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.355922 21222 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.356362 21272 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:08.359812 20846 leader_election.cc:341] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.371366 21301 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.372275 21301 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.376072 21301 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.378321 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.361982 20846 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.379360 21029 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:08.380373 21314 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.382251 20921 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.383553 21270 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.384070 21270 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.384536 21270 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.386149 21072 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.386654 21072 ts_tablet_manager.cc:1367] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.133s user 0.029s sys 0.043s
I20240506 15:14:08.387852 21073 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.388428 21073 ts_tablet_manager.cc:1367] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.144s user 0.014s sys 0.004s
I20240506 15:14:08.393862 21314 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:08.398975 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.404989 21270 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.405527 21072 raft_consensus.cc:348] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.408411 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.411165 21072 raft_consensus.cc:374] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.405508 21073 raft_consensus.cc:348] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.411525 21072 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.411701 21073 raft_consensus.cc:374] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.412132 21073 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.387243 21222 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.412276 21072 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.413350 20877 raft_consensus.cc:2379] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:08.413090 21073 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.413980 21072 ts_tablet_manager.cc:1398] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20240506 15:14:08.414856 21072 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.409873 21074 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.410120 21301 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:08.425267 21074 ts_tablet_manager.cc:1367] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.160s user 0.024s sys 0.011s
I20240506 15:14:08.426842 20957 raft_consensus.cc:2379] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:08.426934 21222 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.428210 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.429271 21029 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.429970 21270 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.435190 21074 raft_consensus.cc:348] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.435979 21074 raft_consensus.cc:374] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.436085 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.422399 21072 tablet_bootstrap.cc:654] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.423566 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:08.436743 21270 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:08.437078 20877 raft_consensus.cc:2379] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:08.436324 21074 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.439127 21074 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.441471 21074 ts_tablet_manager.cc:1398] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240506 15:14:08.436504 21029 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:08.442718 21074 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.443099 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:08.445303 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:08.446311 20957 raft_consensus.cc:2379] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:08.447104 21222 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.449587 21074 tablet_bootstrap.cc:654] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.447413 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:08.453469 21301 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.421018 21073 ts_tablet_manager.cc:1398] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20240506 15:14:08.450886 21222 raft_consensus.cc:686] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:08.455228 21073 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.455035 21222 consensus_queue.cc:237] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.491305 21073 tablet_bootstrap.cc:654] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.490248 21030 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.503728 21072 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.504281 21072 ts_tablet_manager.cc:1367] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.013s
I20240506 15:14:08.504480 21074 tablet_bootstrap.cc:492] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.505096 21074 ts_tablet_manager.cc:1367] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.004s
I20240506 15:14:08.507073 21072 raft_consensus.cc:348] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.508288 21072 raft_consensus.cc:374] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.508581 21072 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.510058 21301 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.504557 20780 catalog_manager.cc:5484] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:08.510798 21301 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.517321 21072 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.518131 21074 raft_consensus.cc:348] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.520072 21074 raft_consensus.cc:374] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.520383 21074 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.520406 21072 ts_tablet_manager.cc:1398] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20240506 15:14:08.521176 21074 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.521703 21072 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.524328 21074 ts_tablet_manager.cc:1398] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.019s user 0.009s sys 0.007s
I20240506 15:14:08.525512 21074 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.526324 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.526928 21301 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.526219 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.527367 21028 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.529893 20877 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
W20240506 15:14:08.527971 20921 leader_election.cc:341] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.533269 21072 tablet_bootstrap.cc:654] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.533592 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:08.538941 21074 tablet_bootstrap.cc:654] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.543227 21301 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.549580 21107 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.550335 21107 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.543991 21301 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.553117 21301 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.554630 21107 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.555454 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.556727 20877 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:08.558532 20924 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.559674 21107 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.560200 21107 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.560684 21107 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.509083 21222 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240506 15:14:08.573096 21107 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.575345 21107 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.577023 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.577719 21029 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.583221 21235 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.583971 21235 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.585954 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:08.586845 20883 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.587836 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.591408 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:08.591475 20998 leader_election.cc:341] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:08.592978 20996 leader_election.cc:341] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.593434 20996 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.594362 21279 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.596330 20883 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:08.595717 21235 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.606608 21074 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.607151 21074 ts_tablet_manager.cc:1367] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.009s
I20240506 15:14:08.608904 21340 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:08.609406 21340 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.610325 21074 raft_consensus.cc:348] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.611065 21074 raft_consensus.cc:374] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.611326 21074 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.613235 21340 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.615512 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.612629 21074 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.598024 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.597923 21029 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:08.617388 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.632588 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.643467 21074 ts_tablet_manager.cc:1398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20240506 15:14:08.645200 21074 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.646600 21340 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:08.656037 21073 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.657035 21073 ts_tablet_manager.cc:1367] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.004s
I20240506 15:14:08.646660 21107 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.661891 21222 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.665098 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.663436 21107 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.671252 21222 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.672076 21353 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.672369 21107 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.672652 21353 raft_consensus.cc:686] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:08.673199 21072 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.673655 21072 ts_tablet_manager.cc:1367] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.152s user 0.099s sys 0.038s
I20240506 15:14:08.673482 21353 consensus_queue.cc:237] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.675736 21222 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.676208 21072 raft_consensus.cc:348] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.678495 21072 raft_consensus.cc:374] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.678825 21072 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.679536 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.682029 21072 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.682240 21031 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:08.684202 21073 raft_consensus.cc:348] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.684772 21072 ts_tablet_manager.cc:1398] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240506 15:14:08.685025 21073 raft_consensus.cc:374] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.685446 21073 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.686086 21073 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.687500 21073 ts_tablet_manager.cc:1398] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.030s user 0.003s sys 0.001s
I20240506 15:14:08.688505 21073 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.691125 21072 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.698807 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.694371 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.703562 21270 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.703979 21270 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.704353 21270 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.725164 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:08.729089 20845 leader_election.cc:341] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.735788 21074 tablet_bootstrap.cc:654] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:08.737740 20845 consensus_peers.cc:485] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:08.738214 21270 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.738451 21301 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.761317 21270 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.764627 21073 tablet_bootstrap.cc:654] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.768791 21222 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.769381 21222 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.771617 21222 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.775835 21072 tablet_bootstrap.cc:654] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.778873 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.779640 20877 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:08.779825 20780 catalog_manager.cc:5484] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:08.780622 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.781363 21028 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:08.783679 21301 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.790922 21279 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.789984 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.791440 21279 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.791782 21029 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.793774 21279 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.794585 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.783851 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:08.802314 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.802886 21222 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.803339 20952 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:08.803587 21222 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.804220 21222 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.806548 20996 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.807642 21235 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.808234 21235 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.808688 21235 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:08.811295 20921 leader_election.cc:341] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:08.816378 20846 leader_election.cc:341] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.816922 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.817863 21313 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.820102 21222 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.822773 21222 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.825673 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:08.826445 20883 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:08.826349 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:08.827006 21074 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.827440 21074 ts_tablet_manager.cc:1367] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.183s user 0.017s sys 0.026s
I20240506 15:14:08.829516 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.831068 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:08.831377 21029 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:08.831987 20952 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:08.833598 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:08.833997 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:08.834746 20957 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:08.836717 21301 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.838523 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:08.839443 20878 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.838901 21074 raft_consensus.cc:348] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.839859 21079 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:08.840209 21074 raft_consensus.cc:374] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.840628 21074 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.841470 21074 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.841835 21301 raft_consensus.cc:686] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:08.842705 21301 consensus_queue.cc:237] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.843822 21235 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.847738 21074 ts_tablet_manager.cc:1398] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20240506 15:14:08.848803 21074 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.848865 20878 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:08.847949 21073 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:08.849786 21073 ts_tablet_manager.cc:1367] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.161s user 0.015s sys 0.001s
W20240506 15:14:08.850718 20998 leader_election.cc:341] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.853726 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:08.854589 21073 raft_consensus.cc:348] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.855072 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.856986 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:08.857625 21079 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.857764 20957 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:08.861531 20998 leader_election.cc:341] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.855044 21074 tablet_bootstrap.cc:654] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.865113 21235 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:08.865571 21222 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:08.855329 21073 raft_consensus.cc:374] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.868623 21072 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:08.868906 21073 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.869608 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:08.869819 21073 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.869117 21072 ts_tablet_manager.cc:1367] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.178s user 0.012s sys 0.005s
I20240506 15:14:08.874190 21074 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:08.874003 21353 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.874715 21353 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:08.875527 21353 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.868141 20957 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:08.877503 20996 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:08.875813 21072 raft_consensus.cc:348] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.878585 21072 raft_consensus.cc:374] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.878943 21072 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.882802 21235 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.883389 21235 raft_consensus.cc:686] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:08.884632 21235 consensus_queue.cc:237] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.892997 21222 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.893498 21074 ts_tablet_manager.cc:1367] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.002s
I20240506 15:14:08.893756 21073 ts_tablet_manager.cc:1398] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20240506 15:14:08.893602 21222 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.879545 21072 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.894660 21073 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:08.901791 21074 raft_consensus.cc:348] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.902451 21074 raft_consensus.cc:374] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:08.902700 21074 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:08.903247 21074 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.896762 21222 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.900630 20777 catalog_manager.cc:5484] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:08.897959 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.908735 21072 ts_tablet_manager.cc:1398] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20240506 15:14:08.909966 21072 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:08.909636 21352 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:08.915902 21352 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.924340 21352 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.913820 21028 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:08.926172 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:08.933470 20846 leader_election.cc:341] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.934238 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:08.935977 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:08.936566 20921 leader_election.cc:341] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.937211 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.938022 21079 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:08.938385 21079 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:08.938711 21079 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:08.941486 20924 leader_election.cc:341] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.941974 20924 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:08.943117 21353 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:08.956568 21079 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:08.958374 20777 catalog_manager.cc:5484] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:08.960024 21074 ts_tablet_manager.cc:1398] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.066s user 0.005s sys 0.003s
I20240506 15:14:08.961122 21074 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:08.956172 21073 tablet_bootstrap.cc:654] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.966069 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:08.967689 21079 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:08.967108 20779 catalog_manager.cc:5484] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
W20240506 15:14:08.970324 20846 leader_election.cc:341] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:08.975973 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:08.976709 21028 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:08.980165 21072 tablet_bootstrap.cc:654] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.980849 21272 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:08.984349 21028 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:08.989694 21074 tablet_bootstrap.cc:654] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:08.996874 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:08.997819 21079 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:08.998612 21079 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:08.999779 21079 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.004065 21073 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.009673 21073 ts_tablet_manager.cc:1367] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.009s
I20240506 15:14:09.018409 21073 raft_consensus.cc:348] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.019322 21073 raft_consensus.cc:374] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.019654 21073 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.020462 21073 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.021768 21074 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.022255 21074 ts_tablet_manager.cc:1367] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20240506 15:14:09.022984 21073 ts_tablet_manager.cc:1398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240506 15:14:09.024201 21073 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.025457 21074 raft_consensus.cc:348] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.026569 21074 raft_consensus.cc:374] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.026854 21074 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
W20240506 15:14:09.027691 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:09.027788 21074 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.032723 21072 tablet_bootstrap.cc:492] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.032963 21074 ts_tablet_manager.cc:1398] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240506 15:14:09.030961 21073 tablet_bootstrap.cc:654] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.033370 21072 ts_tablet_manager.cc:1367] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.124s user 0.029s sys 0.003s
I20240506 15:14:09.034288 21074 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.037808 21072 raft_consensus.cc:348] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.038712 21072 raft_consensus.cc:374] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.039153 21072 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.039997 21072 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.041918 21072 ts_tablet_manager.cc:1398] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240506 15:14:09.044310 21072 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.047240 21074 tablet_bootstrap.cc:654] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.049758 20777 catalog_manager.cc:5484] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.075172 21072 tablet_bootstrap.cc:654] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.076850 21340 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.077752 21340 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.081861 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.086252 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.079972 21340 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:09.090866 20996 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.091810 21235 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.102113 21074 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.103046 21270 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.107328 21074 ts_tablet_manager.cc:1367] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.018s
I20240506 15:14:09.107203 21270 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.110450 21270 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.104578 21073 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.111384 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.111757 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.112447 20878 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
W20240506 15:14:09.113492 20921 leader_election.cc:341] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.114866 20924 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.115993 21270 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.116425 21270 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.116778 21270 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.111426 21073 ts_tablet_manager.cc:1367] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.088s user 0.027s sys 0.015s
I20240506 15:14:09.110487 21074 raft_consensus.cc:348] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.119704 21074 raft_consensus.cc:374] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.120178 21074 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.121062 21074 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.123095 21270 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.126277 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:09.127032 20878 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.130226 21074 ts_tablet_manager.cc:1398] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240506 15:14:09.130906 21073 raft_consensus.cc:348] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.131357 21074 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.131639 21073 raft_consensus.cc:374] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.131959 21073 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.132597 21073 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.136642 21270 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.138907 21074 tablet_bootstrap.cc:654] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.140717 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.134179 20878 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
W20240506 15:14:09.142401 20921 leader_election.cc:341] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.149410 20924 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.152089 21073 ts_tablet_manager.cc:1398] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20240506 15:14:09.153002 21352 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:09.153769 21352 raft_consensus.cc:686] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:09.154913 21352 consensus_queue.cc:237] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.156891 21353 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.157567 21353 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.159828 21353 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.163100 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.169018 20877 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:09.168928 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.170660 20924 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:09.173520 20921 leader_election.cc:341] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.167073 21074 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.174700 21074 ts_tablet_manager.cc:1367] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.014s
I20240506 15:14:09.175361 21353 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.175773 21353 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.176147 21353 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.177708 21074 raft_consensus.cc:348] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.178191 20777 catalog_manager.cc:5484] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.178820 21074 raft_consensus.cc:374] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.179499 21074 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.180363 21074 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.183697 21074 ts_tablet_manager.cc:1398] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20240506 15:14:09.187651 21074 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.193420 21352 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.166951 21028 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.198973 21074 tablet_bootstrap.cc:654] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.204555 21079 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.205142 21079 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.207614 21079 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.209599 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.210318 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.211014 21029 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
W20240506 15:14:09.212038 20846 leader_election.cc:341] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.212646 20845 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.214200 21079 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.214793 21079 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.215308 21079 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.219717 21073 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.222478 21074 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.224469 21353 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.226405 21353 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.226871 21301 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.229350 21079 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.230931 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.232524 21074 ts_tablet_manager.cc:1367] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.008s
I20240506 15:14:09.234427 21079 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.235432 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:09.236737 20846 leader_election.cc:341] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.237026 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.237696 21030 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.241799 21079 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.242426 21079 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:09.243028 20921 leader_election.cc:341] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.244944 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:09.245663 20878 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.245813 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:09.248566 20846 leader_election.cc:341] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.249641 21079 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.252492 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.253861 21074 raft_consensus.cc:348] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:09.254463 20845 leader_election.cc:341] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.254642 21074 raft_consensus.cc:374] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.255000 21074 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.254966 20845 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.250504 21079 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.256647 21079 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.255832 21074 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.258065 21222 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.260215 21074 ts_tablet_manager.cc:1398] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20240506 15:14:09.262136 21074 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.262480 21072 tablet_bootstrap.cc:492] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.262943 21072 ts_tablet_manager.cc:1367] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.219s user 0.017s sys 0.048s
I20240506 15:14:09.266451 21072 raft_consensus.cc:348] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.267336 21072 raft_consensus.cc:374] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.267786 21072 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.268775 21072 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.272226 21072 ts_tablet_manager.cc:1398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240506 15:14:09.273265 21072 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.273677 21352 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.275738 21073 tablet_bootstrap.cc:654] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.277029 21079 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.278273 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.278956 20957 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:09.280053 20846 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:09.282882 21311 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.283453 21311 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.285605 21311 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.293150 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.296679 21074 tablet_bootstrap.cc:654] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:09.297123 20845 leader_election.cc:341] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.302400 21030 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:09.303752 20845 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.305308 21272 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:09.307889 21272 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:09.310041 21352 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.310359 21272 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.318085 21222 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.318606 21222 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.321779 21107 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240506 15:14:09.334331 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:09.339421 20846 leader_election.cc:341] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.343202 21222 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.344409 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:09.347770 20845 leader_election.cc:341] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.348359 20845 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.348830 21079 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.349386 21079 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.350680 21222 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.351783 21079 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.352779 20878 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:09.357158 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:09.364830 20846 leader_election.cc:341] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.366429 20924 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.368176 21107 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:09.374563 21107 raft_consensus.cc:686] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:09.379602 21107 consensus_queue.cc:237] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.387372 21352 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.388243 21352 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.397688 21352 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.398046 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.398744 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.398783 20878 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:09.403074 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:09.403826 20921 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.409456 21072 tablet_bootstrap.cc:654] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.406867 21353 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:09.412210 21353 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.415751 21353 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:09.404076 21311 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.419158 21311 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.422097 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:09.422890 21073 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.423377 21073 ts_tablet_manager.cc:1367] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.024s
W20240506 15:14:09.425320 20921 consensus_peers.cc:485] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240506 15:14:09.429595 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 79812us
W20240506 15:14:09.430797 20998 consensus_peers.cc:485] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e -> Peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Couldn't send request to peer 00fe71c1675441379f5f2c7490f9e63a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:09.433877 21073 raft_consensus.cc:348] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.437117 21073 raft_consensus.cc:374] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.437440 21073 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.438056 21073 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.440618 21073 ts_tablet_manager.cc:1398] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240506 15:14:09.441519 21073 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.422896 20952 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.449756 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.450140 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.449820 21074 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.451383 21074 ts_tablet_manager.cc:1367] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.024s
I20240506 15:14:09.451905 21353 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.462213 21353 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.462746 21279 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.464841 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:09.466184 20921 leader_election.cc:341] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.464883 20780 catalog_manager.cc:5484] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.472097 21074 raft_consensus.cc:348] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.473229 21074 raft_consensus.cc:374] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.473140 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:09.474195 21074 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.473944 20877 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:09.457505 20845 leader_election.cc:341] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.477093 20952 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:09.482064 21074 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:09.485255 20845 leader_election.cc:341] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.487128 20846 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.485746 20845 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.493171 21074 ts_tablet_manager.cc:1398] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.042s user 0.011s sys 0.000s
I20240506 15:14:09.494182 21074 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.496208 20877 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:09.497710 21311 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:09.497833 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.498342 21311 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:09.499289 21107 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:09.499208 21311 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.501052 21107 raft_consensus.cc:686] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:09.502018 21107 consensus_queue.cc:237] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.505813 21073 tablet_bootstrap.cc:654] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.504756 20777 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.508006 21388 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.510147 21072 tablet_bootstrap.cc:492] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.510601 21072 ts_tablet_manager.cc:1367] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.238s user 0.016s sys 0.013s
I20240506 15:14:09.514108 21072 raft_consensus.cc:348] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.515269 21072 raft_consensus.cc:374] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.515640 21072 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.516724 21072 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.522647 21072 ts_tablet_manager.cc:1398] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240506 15:14:09.523528 21072 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.524899 21388 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.525481 21388 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.532682 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.530347 21388 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.535796 20780 catalog_manager.cc:5484] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 2, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.533736 20952 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:09.539441 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.540381 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:09.540139 21032 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:09.541798 21272 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:09.542186 21272 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.542552 21272 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:09.547210 20779 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.549641 21279 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.550371 21279 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.552302 21072 tablet_bootstrap.cc:654] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.554034 21279 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:09.554809 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.555359 21074 tablet_bootstrap.cc:654] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.558072 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.558876 20883 raft_consensus.cc:2379] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:09.559420 21354 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.559937 21354 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.561060 20998 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.562263 21354 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.562871 21340 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.570272 21073 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.570822 21073 ts_tablet_manager.cc:1367] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.005s
I20240506 15:14:09.570504 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.572702 20883 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:09.574326 20924 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:09.577723 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.576822 21107 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.579339 21107 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.579828 21107 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:09.582849 20921 leader_election.cc:341] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.594293 21340 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.597249 21272 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.604180 21354 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:09.608451 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.609549 21030 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:09.608520 21272 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.613102 21072 tablet_bootstrap.cc:492] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.606561 21107 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.613626 21072 ts_tablet_manager.cc:1367] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.090s user 0.017s sys 0.007s
I20240506 15:14:09.614691 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:09.615615 20957 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:09.619962 21073 raft_consensus.cc:348] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:09.620826 20846 consensus_peers.cc:485] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a -> Peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Couldn't send request to peer ecededcd854842d4867fda9fa63a4b66. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:09.620904 21073 raft_consensus.cc:374] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.621580 21073 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.621768 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.622455 21073 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.623243 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:09.623880 20878 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.625674 21074 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
W20240506 15:14:09.626953 20921 leader_election.cc:341] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.628001 21074 ts_tablet_manager.cc:1367] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.002s
I20240506 15:14:09.629209 21107 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.628486 21079 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:09.628693 21030 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:09.634533 20845 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:09.635505 21079 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:09.636451 21079 raft_consensus.cc:686] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:09.636842 20878 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:09.637364 21079 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.638180 21073 ts_tablet_manager.cc:1398] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.067s user 0.008s sys 0.002s
I20240506 15:14:09.645938 21072 raft_consensus.cc:348] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.646760 21072 raft_consensus.cc:374] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.646518 21074 raft_consensus.cc:348] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.647066 21072 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.647279 21074 raft_consensus.cc:374] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.647625 21074 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.648559 21074 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.650739 21074 ts_tablet_manager.cc:1398] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.022s user 0.001s sys 0.005s
I20240506 15:14:09.651831 21074 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.659304 21074 tablet_bootstrap.cc:654] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.659314 20924 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:09.660450 21270 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:09.661032 21270 raft_consensus.cc:686] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:09.661849 21270 consensus_queue.cc:237] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.655584 20957 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:09.668080 21354 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:09.673125 21073 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.647822 21072 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.719841 21072 ts_tablet_manager.cc:1398] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.106s user 0.004s sys 0.003s
I20240506 15:14:09.721021 21072 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.734508 21073 tablet_bootstrap.cc:654] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.735445 21279 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.735934 21279 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.738099 21279 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:09.739027 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.741533 20777 catalog_manager.cc:5484] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.743153 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.745100 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.747828 21340 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.753404 21073 tablet_bootstrap.cc:492] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.756490 21079 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.757001 21079 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.757795 20780 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 2, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 2 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:09.758803 21079 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.764979 21074 tablet_bootstrap.cc:492] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.765122 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.765508 21074 ts_tablet_manager.cc:1367] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.114s user 0.029s sys 0.019s
I20240506 15:14:09.768517 21074 raft_consensus.cc:348] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.769261 21074 raft_consensus.cc:374] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 15:14:09.769644 20846 leader_election.cc:341] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.770382 21074 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.770336 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.771565 21073 ts_tablet_manager.cc:1367] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.013s
W20240506 15:14:09.773820 20845 leader_election.cc:341] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.774407 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.777352 21072 tablet_bootstrap.cc:654] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.774121 21073 raft_consensus.cc:348] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.779825 21074 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.780838 21073 raft_consensus.cc:374] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.781342 21073 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.781997 21073 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.783185 21388 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.784651 21073 ts_tablet_manager.cc:1398] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20240506 15:14:09.785648 21073 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.790767 21074 ts_tablet_manager.cc:1398] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240506 15:14:09.792810 21074 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.793509 21073 tablet_bootstrap.cc:654] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:09.797214 20846 consensus_peers.cc:485] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a -> Peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Couldn't send request to peer ecededcd854842d4867fda9fa63a4b66. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:09.801290 21074 tablet_bootstrap.cc:654] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.802980 21072 tablet_bootstrap.cc:492] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.803341 21073 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.803779 21073 ts_tablet_manager.cc:1367] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20240506 15:14:09.803802 21072 ts_tablet_manager.cc:1367] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.007s
I20240506 15:14:09.806536 21073 raft_consensus.cc:348] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.806632 21072 raft_consensus.cc:348] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.808246 21072 raft_consensus.cc:374] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.808550 21072 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.809305 21072 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.807263 21073 raft_consensus.cc:374] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.810427 21272 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:09.813251 21072 ts_tablet_manager.cc:1398] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240506 15:14:09.814736 21072 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.810859 21073 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.820554 21073 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.828768 21073 ts_tablet_manager.cc:1398] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20240506 15:14:09.829864 21073 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.832608 21072 tablet_bootstrap.cc:654] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.842425 21074 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.843051 21074 ts_tablet_manager.cc:1367] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.017s
I20240506 15:14:09.846719 21074 raft_consensus.cc:348] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.847544 21074 raft_consensus.cc:374] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.848052 21074 raft_consensus.cc:729] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.848814 21074 consensus_queue.cc:260] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.850997 21074 ts_tablet_manager.cc:1398] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240506 15:14:09.852464 21074 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.858812 21074 tablet_bootstrap.cc:654] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.859431 21073 tablet_bootstrap.cc:654] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.870085 21072 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:09.870607 21073 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.873242 21072 ts_tablet_manager.cc:1367] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.017s
I20240506 15:14:09.873275 21073 ts_tablet_manager.cc:1367] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.001s
I20240506 15:14:09.871342 21074 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.874984 21074 ts_tablet_manager.cc:1367] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20240506 15:14:09.875893 21072 raft_consensus.cc:348] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.876662 21072 raft_consensus.cc:374] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.876926 21072 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.877799 21074 raft_consensus.cc:348] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.878748 21074 raft_consensus.cc:374] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.879091 21074 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.877882 21072 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.881542 21074 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.884318 21074 ts_tablet_manager.cc:1398] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240506 15:14:09.885542 21074 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.881496 21073 raft_consensus.cc:348] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.891326 21073 raft_consensus.cc:374] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.891660 21073 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.891786 21074 tablet_bootstrap.cc:654] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.892395 21073 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.893632 21072 ts_tablet_manager.cc:1398] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20240506 15:14:09.894554 21073 ts_tablet_manager.cc:1398] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240506 15:14:09.894562 21072 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:09.895910 21073 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:09.898178 21030 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.901027 21072 tablet_bootstrap.cc:654] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.902628 21396 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.903163 21396 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.907836 21074 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:09.908450 21074 ts_tablet_manager.cc:1367] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20240506 15:14:09.910547 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.911496 20877 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:09.910975 21074 raft_consensus.cc:348] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.912348 21074 raft_consensus.cc:374] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:09.912685 21074 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:09.913425 21074 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.913848 21353 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.915755 21074 ts_tablet_manager.cc:1398] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240506 15:14:09.916689 21074 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:09.919080 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.920588 21073 tablet_bootstrap.cc:654] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.922000 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:09.922909 21074 tablet_bootstrap.cc:654] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:09.923498 21340 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.923621 21353 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.932721 21353 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.933774 21396 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:09.934913 21353 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.935833 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.936810 20878 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:09.938378 20924 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:09.940608 21270 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:09.941027 21270 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:09.941500 21270 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.942978 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:09.940898 21340 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:09.947921 21340 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.948455 21340 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:09.950270 20921 leader_election.cc:341] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.952410 21270 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.956410 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:09.957363 21340 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:09.958424 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:09.959563 20878 raft_consensus.cc:2354] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2.
I20240506 15:14:09.961441 20998 leader_election.cc:398] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2.
I20240506 15:14:09.962311 20998 leader_election.cc:401] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:09.967624 21079 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:09.968227 21079 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:09.970175 21079 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.971277 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:09.971661 20921 leader_election.cc:341] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:09.972986 20845 leader_election.cc:341] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:09.976771 21270 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:09.978103 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:09.980115 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:09.980795 20878 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:09.981029 21073 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:09.981477 21073 ts_tablet_manager.cc:1367] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.000s
I20240506 15:14:09.997073 21396 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 2
W20240506 15:14:09.997133 20921 consensus_peers.cc:485] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:10.008409 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.010201 21079 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.010932 21079 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.013820 21079 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.009025 20877 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:10.019937 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:10.018746 21107 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.021301 21107 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.010100 20957 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:10.027724 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:10.036588 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:10.037274 20883 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:10.038036 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.041003 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:10.042878 21079 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:10.043236 21079 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:10.043529 21079 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:10.053567 20921 leader_election.cc:341] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:10.053822 21079 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.024109 21107 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.059486 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:10.061782 20845 leader_election.cc:341] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:10.062732 21079 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.066947 21072 tablet_bootstrap.cc:492] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:10.067466 21072 ts_tablet_manager.cc:1367] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.173s user 0.037s sys 0.031s
I20240506 15:14:10.069295 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:10.070245 20953 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.070883 21072 raft_consensus.cc:348] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.071732 21072 raft_consensus.cc:374] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.072024 21072 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.072870 21072 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.081192 21072 ts_tablet_manager.cc:1398] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240506 15:14:10.083510 21422 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:10.083549 21074 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:10.082233 21072 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:10.084749 21074 ts_tablet_manager.cc:1367] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.168s user 0.046s sys 0.064s
I20240506 15:14:10.087129 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:10.088042 21074 raft_consensus.cc:348] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.088900 21074 raft_consensus.cc:374] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.089231 21074 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.090802 21072 tablet_bootstrap.cc:654] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:10.090029 21074 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.088076 21107 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:10.094987 21107 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:10.095149 21074 ts_tablet_manager.cc:1398] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20240506 15:14:10.095489 21107 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.096418 21074 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:10.100494 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:10.101747 21079 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:10.126515 21079 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:09.983928 21073 raft_consensus.cc:348] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.132988 21073 raft_consensus.cc:374] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 15:14:10.133788 20845 consensus_peers.cc:485] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:10.138039 21396 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2.
I20240506 15:14:10.133838 21073 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.141857 20953 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:10.142784 21032 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:10.141956 21107 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.164027 21107 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.165243 20954 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:10.165576 21073 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.143824 20878 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:10.168525 21421 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:10.168715 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:10.169425 20924 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:10.152732 21072 tablet_bootstrap.cc:492] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:10.155560 20877 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.170449 21072 ts_tablet_manager.cc:1367] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.088s user 0.032s sys 0.020s
I20240506 15:14:10.170460 21079 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:10.171550 21079 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:10.172379 21079 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.173339 21072 raft_consensus.cc:348] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.173992 21072 raft_consensus.cc:374] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.174250 21072 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.175164 21072 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.145731 21074 tablet_bootstrap.cc:654] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:10.142642 20957 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.178880 21422 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:10.179395 21422 raft_consensus.cc:686] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:10.180235 21422 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.167502 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:10.184682 21072 ts_tablet_manager.cc:1398] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240506 15:14:10.185721 21072 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:10.186656 21270 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.187619 21270 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.189486 21235 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.190050 21235 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.191751 21353 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:10.193029 21235 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:10.195443 21270 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.207309 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:10.208371 20879 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.226415 21073 ts_tablet_manager.cc:1398] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.245s user 0.013s sys 0.000s
I20240506 15:14:10.195523 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:10.232597 20883 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:10.233923 20998 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:10.235656 21340 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:10.236040 21340 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:10.236440 21340 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.239601 21079 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.240316 21235 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.240453 21079 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.240815 21235 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.241993 21073 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:10.244894 21074 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:10.246222 21235 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:10.248054 21079 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.249576 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:10.251888 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:10.259637 21029 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20240506 15:14:10.263159 20845 leader_election.cc:341] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:10.264075 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 94740us
I20240506 15:14:10.253078 21072 tablet_bootstrap.cc:654] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:10.256759 21074 ts_tablet_manager.cc:1367] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.161s user 0.010s sys 0.003s
I20240506 15:14:10.265859 20879 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:10.275669 21272 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240506 15:14:10.283681 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:10.315097 21079 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:10.327283 20779 catalog_manager.cc:5484] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:10.338029 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 65568us
I20240506 15:14:10.244086 21340 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.345228 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:10.355060 21107 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
W20240506 15:14:10.365094 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 182553us
I20240506 15:14:10.347384 21405 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.374500 21107 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:10.351289 20878 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:10.378664 21107 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:10.444926 20921 leader_election.cc:341] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:10.420738 21270 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.415256 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.399932 21405 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.469478 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.471556 20958 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:10.479009 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:10.484809 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:10.470763 21354 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:10.490204 21340 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:10.491252 21074 raft_consensus.cc:348] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.492801 21074 raft_consensus.cc:374] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.493368 21074 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.494915 21074 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.517858 21074 ts_tablet_manager.cc:1398] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.252s user 0.010s sys 0.002s
I20240506 15:14:10.487829 21353 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:10.472482 21270 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.510922 21436 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.528002 21353 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:10.548822 21432 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
W20240506 15:14:10.549927 20846 leader_election.cc:341] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:10.550407 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:10.549456 21405 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.552908 21439 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:10.559300 21073 tablet_bootstrap.cc:654] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:10.560434 21436 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.501559 21354 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.562394 21353 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:10.583024 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 99164.1us
I20240506 15:14:10.584494 20954 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:10.582185 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:10.597507 21270 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:10.636471 20845 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:10.650887 21436 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.684684 21028 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:10.687454 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 87357.5us
I20240506 15:14:10.689373 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:10.660593 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:10.725350 21074 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:10.677393 21270 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20240506 15:14:10.726961 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:10.788686 21072 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:10.789278 21072 ts_tablet_manager.cc:1367] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.604s user 0.008s sys 0.015s
I20240506 15:14:10.738179 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.837996 20777 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:10.851156 21072 raft_consensus.cc:348] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.855193 21072 raft_consensus.cc:374] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.846072 21439 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:10.855926 21072 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.860888 21072 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.810483 21353 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.857887 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:10.858719 21406 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.007s sys 0.009s
I20240506 15:14:10.872236 20957 raft_consensus.cc:2379] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:10.872746 21072 ts_tablet_manager.cc:1398] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.083s user 0.008s sys 0.002s
W20240506 15:14:10.870851 21406 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s user 0.008s sys 0.009s
I20240506 15:14:10.864287 21439 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.886485 21073 tablet_bootstrap.cc:492] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:10.886982 21073 ts_tablet_manager.cc:1367] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.645s user 0.030s sys 0.047s
I20240506 15:14:10.896939 21073 raft_consensus.cc:348] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.898171 21073 raft_consensus.cc:374] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:10.903422 21073 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:10.878187 21353 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.905534 21073 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:10.889042 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:10.929971 21074 tablet_bootstrap.cc:654] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:10.964494 21354 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:10.967607 20921 consensus_peers.cc:485] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 15:14:10.977274 20921 consensus_peers.cc:485] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:10.930836 21439 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:10.928340 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:10.989879 21073 ts_tablet_manager.cc:1398] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.103s user 0.010s sys 0.003s
I20240506 15:14:10.926946 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.013186 21073 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
W20240506 15:14:11.016769 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 152608us
I20240506 15:14:11.008085 21025 raft_consensus.cc:2379] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:11.010465 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.019207 21353 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:11.020890 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.150005 20950 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:11.177459 21460 log.cc:924] Time spent T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20240506 15:14:11.184746 21030 raft_consensus.cc:2379] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:11.056536 21353 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.272086 21073 tablet_bootstrap.cc:654] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:11.279218 21472 log.cc:924] Time spent T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
W20240506 15:14:11.299566 21475 log.cc:924] Time spent T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20240506 15:14:11.300664 20954 consensus_meta.cc:287] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.697s user 0.009s sys 0.004s
W20240506 15:14:11.298554 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 148708us
I20240506 15:14:11.333624 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:11.334350 20883 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:11.387529 21455 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.388312 21455 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.391695 21455 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.379710 20883 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:11.397101 21353 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:11.399782 21486 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.400604 21486 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.412196 21455 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.509085 21455 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.515020 20998 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:11.515908 21396 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:11.516424 21396 raft_consensus.cc:686] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:11.481503 21235 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.517395 21396 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.523569 21235 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.532277 21235 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.535789 21074 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:11.536163 21455 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.536372 21074 ts_tablet_manager.cc:1367] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.811s user 0.017s sys 0.020s
I20240506 15:14:11.578011 21493 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:11.580336 21486 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.587334 21486 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.593076 21443 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:11.591528 21486 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.595621 21073 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:11.596108 21073 ts_tablet_manager.cc:1367] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.583s user 0.049s sys 0.101s
W20240506 15:14:11.597106 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 211082us
I20240506 15:14:11.613535 21486 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.620695 21428 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:11.621497 21428 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.582599 21493 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.625939 21493 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.622342 21073 raft_consensus.cc:348] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.631338 21073 raft_consensus.cc:374] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:11.631739 21073 raft_consensus.cc:729] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:11.632496 21073 consensus_queue.cc:260] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.634934 21073 ts_tablet_manager.cc:1398] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.039s user 0.003s sys 0.003s
I20240506 15:14:11.635154 21428 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:11.637961 20996 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:11.639762 21496 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:11.635529 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.643729 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.635947 21073 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:11.645146 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.647866 21072 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:11.668949 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 174) took 1014 ms. Trace:
I20240506 15:14:11.669459 21028 rpcz_store.cc:276] 0506 15:14:10.654629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:10.657651 (+ 3022us) service_pool.cc:225] Handling call
0506 15:14:10.684498 (+ 26847us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:10.901254 (+216756us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:10.901302 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:10.901972 (+ 670us) log.cc:842] Serialized 26 byte log entry
0506 15:14:11.015986 (+114014us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:11.341351 (+325365us) log.cc:842] Serialized 14 byte log entry
0506 15:14:11.371419 (+ 30068us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:11.374239 (+ 2820us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:11.667595 (+293356us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b5400076fa0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e27d9b1 00007f630e288470 00007f630e288418 00007f630e288398 00007f630e288353 00007f630e2875dc 00007f63142f7fd6 00007f63142f05d8 00007f630e2c45db 00007f630e2c70d0
0506 15:14:11.668593 (+ 998us) raft_consensus.cc:1671] finished
0506 15:14:11.668706 (+ 113us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:11.668875 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":168801,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":44,"raft.queue_time_us":18346,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":2082,"spinlock_wait_cycles":4521216,"thread_start_us":395555,"threads_started":2,"wal-append.queue_time_us":142776}
I20240506 15:14:11.685783 21449 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.686434 21449 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.689815 21449 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:11.694556 21491 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.119s user 0.003s sys 0.000s
I20240506 15:14:11.696532 21073 tablet_bootstrap.cc:654] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:11.712867 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.714700 21072 tablet_bootstrap.cc:654] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:11.717737 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.718703 21496 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 15:14:11.719149 21490 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.178s user 0.001s sys 0.001s
I20240506 15:14:11.720357 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:11.724066 21495 log.cc:924] Time spent T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.148s user 0.003s sys 0.000s
I20240506 15:14:11.726004 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.726408 21453 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.727059 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.727218 21453 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.727632 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.727471 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.728538 20958 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:11.729663 21493 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.729838 21453 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.730230 21493 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.716053 21421 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.731974 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.727972 21026 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:11.732688 21421 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.735720 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.736585 21073 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:11.740360 21073 ts_tablet_manager.cc:1367] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.006s
I20240506 15:14:11.740754 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 170) took 1326 ms. Trace:
I20240506 15:14:11.743145 21073 raft_consensus.cc:348] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.741230 20954 rpcz_store.cc:276] 0506 15:14:10.414730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:10.422480 (+ 7750us) service_pool.cc:225] Handling call
0506 15:14:10.481776 (+ 59296us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:11.329303 (+847527us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:11.333340 (+ 4037us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:11.379858 (+ 46518us) log.cc:842] Serialized 26 byte log entry
0506 15:14:11.541056 (+161198us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:11.582476 (+ 41420us) log.cc:842] Serialized 14 byte log entry
0506 15:14:11.590586 (+ 8110us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:11.710265 (+119679us) spinlock_profiling.cc:243] Waited 78.5 ms on lock 0x7b540030a554. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:11.710316 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:11.718581 (+ 8265us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:11.734499 (+ 15918us) raft_consensus.cc:1671] finished
0506 15:14:11.734517 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:11.740689 (+ 6172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20067,"prepare.queue_time_us":28501,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":28754,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":11475,"spinlock_wait_cycles":265960576,"thread_start_us":68969,"threads_started":2,"wal-append.queue_time_us":148692}
I20240506 15:14:11.744017 21073 raft_consensus.cc:374] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:11.744889 21421 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:11.743861 21493 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.745949 21073 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:11.747112 21073 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.748884 21453 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.750658 21453 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.752177 21073 ts_tablet_manager.cc:1398] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240506 15:14:11.753347 21453 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:11.753844 21073 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:11.755429 21072 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:11.755985 21072 ts_tablet_manager.cc:1367] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.112s user 0.026s sys 0.004s
I20240506 15:14:11.811012 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:11.817637 21073 tablet_bootstrap.cc:654] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:11.812502 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 77451.9us
I20240506 15:14:11.817303 21072 raft_consensus.cc:348] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.823827 21072 raft_consensus.cc:374] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:11.819746 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.827167 21072 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:11.828377 21450 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.827926 21444 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:11.831681 21444 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:11.832131 21444 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:11.834162 21072 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.839072 21464 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:11.840199 21464 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.843659 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:11.829553 21450 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.852617 21464 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:11.856525 20777 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 97) took 1459 ms. Trace:
I20240506 15:14:11.857203 21279 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:11.858053 21279 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:11.858598 21279 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:11.861694 21450 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:11.862552 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 109) took 1565 ms. Trace:
I20240506 15:14:11.866948 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.869096 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:11.870431 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:11.871409 21450 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:11.872375 21450 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.882613 21072 ts_tablet_manager.cc:1398] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.126s user 0.011s sys 0.005s
I20240506 15:14:11.883059 21450 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:11.884835 20921 leader_election.cc:341] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:11.884991 21072 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:11.890288 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:11.892021 21026 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:11.862888 20779 rpcz_store.cc:276] 0506 15:14:10.311797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:10.313717 (+ 1920us) service_pool.cc:225] Handling call
0506 15:14:11.862369 (+1548652us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:10.552713 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:10.553258 (+ 545us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:10.553294 (+ 36us) write_op.cc:435] Acquired schema lock
0506 15:14:10.553311 (+ 17us) tablet.cc:660] Decoding operations
0506 15:14:10.553787 (+ 476us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:10.553861 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:10.553888 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:10.554096 (+ 208us) tablet.cc:699] Row locks acquired
0506 15:14:10.554114 (+ 18us) write_op.cc:260] PREPARE: finished
0506 15:14:10.554248 (+ 134us) write_op.cc:270] Start()
0506 15:14:10.554346 (+ 98us) write_op.cc:276] Timestamp: P: 1715008450554208 usec, L: 0
0506 15:14:10.554365 (+ 19us) op_driver.cc:331] REPLICATION: starting
0506 15:14:10.555083 (+ 718us) log.cc:842] Serialized 540 byte log entry
0506 15:14:10.672461 (+117378us) op_driver.cc:444] REPLICATION: finished
0506 15:14:10.809308 (+136847us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630b3a716e 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f6311d00dd0
0506 15:14:11.014747 (+205439us) write_op.cc:301] APPLY: starting
0506 15:14:11.019252 (+ 4505us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:11.030513 (+ 11261us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:11.037152 (+ 6639us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:11.048303 (+ 11151us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:11.048552 (+ 249us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:11.048629 (+ 77us) write_op.cc:312] APPLY: finished
0506 15:14:11.049135 (+ 506us) log.cc:842] Serialized 22 byte log entry
0506 15:14:11.049371 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:11.049444 (+ 73us) write_op.cc:454] Released schema lock
0506 15:14:11.049480 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:11.053842 (+ 4362us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:11.067436 (+ 13594us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":259858,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":303835,"apply.run_cpu_time_us":3318,"apply.run_wall_time_us":137143,"num_ops":1,"prepare.queue_time_us":197609,"prepare.run_cpu_time_us":2773,"prepare.run_wall_time_us":13029,"replication_time_us":105203,"spinlock_wait_cycles":4811008,"thread_start_us":363457,"threads_started":2}]]}
I20240506 15:14:11.894670 21073 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:11.895606 21073 ts_tablet_manager.cc:1367] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.142s user 0.019s sys 0.014s
I20240506 15:14:11.898917 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:11.900193 21421 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:11.900700 21421 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:11.901113 21421 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:11.901947 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 181) took 1401 ms. Trace:
I20240506 15:14:11.902467 21074 raft_consensus.cc:348] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.903697 21074 raft_consensus.cc:374] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:11.904088 21074 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:11.857131 20777 rpcz_store.cc:276] 0506 15:14:10.412621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:10.415676 (+ 3055us) service_pool.cc:225] Handling call
0506 15:14:11.856205 (+1440529us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:10.854918 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:10.900561 (+ 45643us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:10.900594 (+ 33us) write_op.cc:435] Acquired schema lock
0506 15:14:10.900614 (+ 20us) tablet.cc:660] Decoding operations
0506 15:14:10.901003 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:10.901062 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:10.901092 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:10.901269 (+ 177us) tablet.cc:699] Row locks acquired
0506 15:14:10.901281 (+ 12us) write_op.cc:260] PREPARE: finished
0506 15:14:10.901412 (+ 131us) write_op.cc:270] Start()
0506 15:14:10.901514 (+ 102us) write_op.cc:276] Timestamp: P: 1715008450901371 usec, L: 0
0506 15:14:10.901529 (+ 15us) op_driver.cc:331] REPLICATION: starting
0506 15:14:10.961535 (+ 60006us) log.cc:842] Serialized 540 byte log entry
0506 15:14:10.986978 (+ 25443us) op_driver.cc:444] REPLICATION: finished
0506 15:14:11.084943 (+ 97965us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54000037d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eb3e8 00007f6311d0049e 00007f6311cffc78 00007f6311d04561 00007f6311d04519 00007f6311d044a9 00007f6311d04471 00007f6311d0376d 00007f6314639da4 00007f6314639bd9
0506 15:14:11.191788 (+106845us) write_op.cc:301] APPLY: starting
0506 15:14:11.194690 (+ 2902us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:11.249714 (+ 55024us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:11.254605 (+ 4891us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:11.273417 (+ 18812us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:11.279876 (+ 6459us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:11.292405 (+ 12529us) write_op.cc:312] APPLY: finished
0506 15:14:11.319682 (+ 27277us) log.cc:842] Serialized 22 byte log entry
0506 15:14:11.329971 (+ 10289us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:11.336500 (+ 6529us) write_op.cc:454] Released schema lock
0506 15:14:11.340185 (+ 3685us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:11.367576 (+ 27391us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:11.371267 (+ 3691us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":253131,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":195326,"apply.run_cpu_time_us":2967,"apply.run_wall_time_us":183196,"num_ops":1,"prepare.queue_time_us":5297,"prepare.run_cpu_time_us":6878,"prepare.run_wall_time_us":239286,"replication_time_us":79340,"spinlock_wait_cycles":118081280,"thread_start_us":23839,"threads_started":1}]]}
I20240506 15:14:11.938143 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.928170 21405 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:11.962376 21464 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:11.962870 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:11.954224 21405 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.963475 21464 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.966039 21079 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:11.968243 21448 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:11.969569 21388 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.969475 21448 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.970831 21406 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:11.999575 21406 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.988382 21272 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:11.997507 20845 leader_election.cc:341] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:11.998162 21439 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.016204 21272 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.034974 21493 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.035780 21493 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.047353 21074 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.967320 21079 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:11.902432 20878 rpcz_store.cc:276] 0506 15:14:10.498605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:10.509312 (+ 10707us) service_pool.cc:225] Handling call
0506 15:14:10.884916 (+375604us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b58002c4998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d6dc 00007f630b38d29b 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:10.885097 (+ 181us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:10.885291 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 1
0506 15:14:10.983286 (+ 97995us) log.cc:842] Serialized 14 byte log entry
0506 15:14:11.686681 (+703395us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b1800276f40. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3c52e4 00007f630b3c8691 00007f630b3c8649 00007f630b3c85d9 00007f630b3c85a1 00007f630b3c789d 00007f6314639da4 00007f6314639bd9 00007f630b3a7936
0506 15:14:11.888821 (+202140us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b1800276f40. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b33e2ea 00007f6314a2dfe5 00007f630b3c23d4 00007f630b3c1cc5 00007f630e2e7620 00007f630e2f97c6 00007f630e30a327 00007f630e30a2a8 00007f630e30a228
0506 15:14:11.899415 (+ 10594us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:11.899520 (+ 105us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:11.899616 (+ 96us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:11.899734 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:11.899970 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59899,"prepare.queue_time_us":126987,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":1514,"raft.queue_time_us":920224,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":25919,"spinlock_wait_cycles":13354624,"thread_start_us":54710,"threads_started":1}
I20240506 15:14:12.064751 21439 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.052325 21450 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.066107 21388 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.066670 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.078325 21450 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.080785 21388 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.080960 21450 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.083312 21388 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.084589 21388 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.084054 21493 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.070155 21272 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.088304 21388 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.089228 21074 ts_tablet_manager.cc:1398] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.553s user 0.005s sys 0.004s
I20240506 15:14:12.072978 21464 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.094779 21079 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:12.091982 21074 ts_tablet_manager.cc:1430] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Tablet startup took 555ms
I20240506 15:14:12.094892 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.095693 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.098680 21496 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.099443 21496 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.101173 21279 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.087477 21406 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:12.098701 21074 ts_tablet_manager.cc:1432] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Trace:
0506 15:14:03.508171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:03.653403 (+145232us) service_pool.cc:225] Handling call
0506 15:14:03.657023 (+ 3620us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 15:14:03.657133 (+ 110us) ts_tablet_manager.cc:655] Creating new metadata...
0506 15:14:03.716711 (+ 59578us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:03.766362 (+ 49651us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:03.796772 (+ 30410us) inbound_call.cc:173] Queueing success response
0506 15:14:10.724917 (+6928145us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0506 15:14:11.536656 (+811739us) ts_tablet_manager.cc:1399] Starting tablet replica
0506 15:14:11.537370 (+ 714us) tablet_replica.cc:259] Starting instrumentation
0506 15:14:11.537933 (+ 563us) tablet_replica.cc:278] Starting consensus
Metrics: {"mutex_wait_us":368817,"raft.queue_time_us":4952,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"tablet-open.queue_time_us":6950253}
I20240506 15:14:12.113149 21074 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:12.115254 21448 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.121850 21073 raft_consensus.cc:348] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.122869 21073 raft_consensus.cc:374] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:12.122946 21340 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.123203 21073 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:12.123728 21340 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.123961 21073 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.102339 21496 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.128232 21272 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.129230 21272 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.130422 21505 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.131162 21505 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:12.125711 20921 leader_election.cc:341] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.136353 21448 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.137111 21448 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.131772 21272 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:12.147049 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 65815.8us
W20240506 15:14:12.149523 20846 leader_election.cc:341] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.153019 21073 ts_tablet_manager.cc:1398] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.257s user 0.011s sys 0.018s
I20240506 15:14:12.132854 21504 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.151562 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.158919 21506 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.159806 21506 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.162706 21506 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.164944 21448 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.165469 21340 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.158121 21388 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.166798 21279 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.166515 21388 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.167239 21439 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.150990 21405 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.141431 21421 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.156133 21079 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.154626 21272 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.180023 21073 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:12.152585 21444 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.182538 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.184195 21436 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.171025 21505 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.191531 21079 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.178256 21457 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.184011 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.178517 21506 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.186089 21448 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 15:14:12.207368 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 58087.1us
I20240506 15:14:12.178200 21504 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.195655 21439 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.185231 21272 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.199671 21388 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.184978 21436 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.217756 21079 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.199023 21457 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.213910 21439 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.215037 21506 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.235359 21444 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.234198 21079 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.239704 21463 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.230935 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.236570 21508 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.232465 21421 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.235975 21448 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.245591 21448 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.247735 21448 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.248320 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.249058 21496 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:12.241097 21463 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.253111 21510 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:12.252920 21508 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.258608 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:12.259637 20950 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:12.260098 21508 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.259680 21510 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.262074 21457 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.262238 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:12.263178 21029 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:12.263962 21510 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.265228 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:12.265852 21439 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.265975 21026 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.266588 21504 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:12.267689 20845 leader_election.cc:341] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.270381 21507 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
W20240506 15:14:12.273378 20846 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.273893 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.276008 21505 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.241367 21272 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.252171 21463 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.280270 21079 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.283021 21436 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.278659 21072 tablet_bootstrap.cc:654] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:12.300776 20846 leader_election.cc:341] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.301275 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.306620 21079 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.309032 21479 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.305127 21436 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.309862 21479 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.312906 21436 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.314558 21479 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.314926 21436 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.309887 21444 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.317517 21444 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.319316 21444 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.319607 20777 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:12.321077 21388 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.311122 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.322830 21388 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.323565 21388 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.329420 21079 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.330358 21388 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.330910 21026 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:12.330116 21079 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.337275 21444 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.337419 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.339195 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:12.341197 21079 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.340688 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.344892 21479 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.344944 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.345813 21032 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:12.345947 20877 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:12.346027 21444 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.345834 21479 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.348908 21487 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:12.349637 21487 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:12.351065 21073 tablet_bootstrap.cc:654] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:12.352931 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:12.355351 21446 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.355763 21446 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.356199 21446 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.362020 21279 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.362710 21279 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.365267 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.366052 21032 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:12.366586 21074 tablet_bootstrap.cc:654] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:12.368170 21388 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.351152 21487 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.367964 21279 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.362301 21444 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.369066 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:12.371220 21506 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.375870 21444 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.385512 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.378957 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.388036 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.385607 21444 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.388898 20878 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:12.389005 21444 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.391610 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.400882 20877 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:12.401607 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.392483 21479 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.396057 21506 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.412012 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.413166 20883 raft_consensus.cc:2379] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:12.414638 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.417029 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.422894 21506 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.427800 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.428637 20877 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:12.428902 21506 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.429553 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.430182 21072 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:12.430451 20879 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:12.430783 21072 ts_tablet_manager.cc:1367] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.546s user 0.013s sys 0.008s
I20240506 15:14:12.380558 21388 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.433820 21479 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.444470 21446 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.447805 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.449079 21388 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.444470 21479 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.448335 21446 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.453505 21479 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.460881 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.461838 21074 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:12.487623 21074 ts_tablet_manager.cc:1367] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.375s user 0.015s sys 0.014s
I20240506 15:14:12.478508 21072 raft_consensus.cc:348] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.484338 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.490234 21025 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:12.491734 20921 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:12.491643 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.489008 21072 raft_consensus.cc:374] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:12.493541 21072 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:12.494899 21479 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:12.484946 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.495404 21479 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.496050 21479 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:12.496745 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.498420 21072 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.471318 21388 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.483803 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.502638 21388 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.474059 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:12.504516 21032 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.505945 21073 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:12.506475 21073 ts_tablet_manager.cc:1367] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.327s user 0.010s sys 0.024s
I20240506 15:14:12.506055 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.510097 20879 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:12.511253 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.471063 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.470865 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.513258 21487 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.515703 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.515736 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.506152 21444 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.506657 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.517961 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.529793 21444 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.529625 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.496052 20876 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:12.536628 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:12.537241 21025 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.525672 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.525728 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.532120 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.525594 21388 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.532896 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.533221 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.533826 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.534067 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.535521 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.534686 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.527814 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.537173 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.540442 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.541945 21074 raft_consensus.cc:348] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.540055 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.544930 21074 raft_consensus.cc:374] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:12.521560 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.545625 21074 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:12.547574 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.547992 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.556105 20877 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:12.557024 21074 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.564247 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.548468 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.518309 21487 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.548168 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.550215 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.564143 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.564858 21074 ts_tablet_manager.cc:1398] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20240506 15:14:12.549028 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.548542 21032 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:12.569519 21421 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.565991 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.569999 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.572461 20954 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:12.573206 21026 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:12.569919 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.576663 21031 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:12.575392 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.578400 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:12.580696 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.582520 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.583436 21457 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.583294 21484 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:12.585498 20998 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.583932 21457 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.587308 21457 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.564981 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.570271 21421 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.588722 21444 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:12.600127 21074 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:12.594410 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.601200 21444 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:12.595216 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.573199 21388 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.596930 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.598896 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.613900 20998 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.599444 21072 ts_tablet_manager.cc:1398] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.168s user 0.014s sys 0.007s
I20240506 15:14:12.599824 21421 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.602160 21444 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.604911 21340 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.607473 21493 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.621511 21493 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.626710 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.628357 21340 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.631309 21487 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.631783 21340 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.632479 21450 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.632700 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.617201 21388 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.634826 21072 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:12.651955 21496 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.653573 21496 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.654137 21496 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:12.657862 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 44096us
I20240506 15:14:12.659291 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:12.660516 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70284.3us
I20240506 15:14:12.660763 21279 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.662041 20998 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.653051 21493 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.612493 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.676111 21340 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.677927 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.678891 21487 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:12.684511 21073 raft_consensus.cc:348] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.686702 21073 raft_consensus.cc:374] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:12.687026 21073 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:12.688419 21073 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.684042 21388 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.685653 21279 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.690948 21450 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.696369 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.698529 21073 ts_tablet_manager.cc:1398] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.191s user 0.007s sys 0.014s
I20240506 15:14:12.703648 21457 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.698608 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.693940 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.718055 21487 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.726424 21487 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.728633 21487 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.729172 21487 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:12.729600 21487 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.730568 21025 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:12.731719 21487 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.738039 21073 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
W20240506 15:14:12.744778 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 50369us
I20240506 15:14:12.737327 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:12.760082 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 65193.2us
I20240506 15:14:12.733785 21479 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.768044 21457 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.772511 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.758153 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.777596 21340 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.788408 20998 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.788875 21279 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.793623 20924 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.808997 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.811419 21074 tablet_bootstrap.cc:654] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:12.799729 21456 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.788924 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.790962 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:12.819713 20996 leader_election.cc:341] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.817178 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.819373 21526 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.822212 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:12.832859 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:12.833964 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.834844 21479 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.835503 21073 tablet_bootstrap.cc:654] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:12.835543 20924 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.837683 21450 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:12.837743 21430 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.838174 21450 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:12.838768 21450 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:12.839625 20924 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.841271 21072 tablet_bootstrap.cc:654] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:12.842022 20924 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.841399 21444 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.842962 21526 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:12.844940 20924 leader_election.cc:341] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:12.845312 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.845378 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.845835 21496 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.846338 21493 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.846768 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.846004 21526 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.846868 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.847708 21493 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:12.847826 21444 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.848176 21493 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:12.848930 21493 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:12.849876 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.849574 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.848023 21496 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:12.850693 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:12.847980 20998 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.852281 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.851449 21444 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.852927 21496 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:12.853201 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.840888 20877 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:12.853930 21496 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.853907 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.854565 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.855405 21496 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:12.856086 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.852027 21524 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.857314 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.858929 21524 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.859534 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:12.860056 20998 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.860328 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.861379 20998 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:12.863633 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:12.864073 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:12.864845 21526 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:12.865378 21526 raft_consensus.cc:686] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:12.865628 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.865895 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.866207 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.866811 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.867497 20949 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:12.867609 20954 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:12.871906 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.868681 21496 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:12.866290 21526 consensus_queue.cc:237] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.869431 21524 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:12.869805 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.869155 21479 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.870271 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.874646 21479 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.878211 21444 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.877429 21525 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:12.878949 21444 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.879470 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:12.879779 21525 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.879231 20950 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240506 15:14:12.882339 21444 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.883137 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.886240 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.889987 21525 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.889900 21450 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:12.894568 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.895136 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.895640 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.896186 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.895954 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.896458 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.897817 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.898353 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.898571 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.896548 21031 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:12.906435 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.906879 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:12.907109 21029 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:12.906157 21479 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.912849 21450 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:12.945773 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:12.978181 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.008602 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.009307 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.018400 21074 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:13.018965 21074 ts_tablet_manager.cc:1367] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.429s user 0.009s sys 0.011s
I20240506 15:14:13.019306 21073 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:13.019809 21073 ts_tablet_manager.cc:1367] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.306s user 0.007s sys 0.019s
I20240506 15:14:13.022128 21074 raft_consensus.cc:348] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.023216 21074 raft_consensus.cc:374] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.023569 21074 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.030753 21074 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.040398 21079 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:13.040755 20845 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.041036 21493 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.041923 21079 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.042251 21079 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.043148 21493 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:13.042984 21079 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.043751 21032 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.044553 21448 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.044186 21079 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.045760 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.047478 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.047772 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.048301 21072 tablet_bootstrap.cc:492] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
W20240506 15:14:13.048686 20998 leader_election.cc:341] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.048982 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.049883 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:13.048729 21072 ts_tablet_manager.cc:1367] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.414s user 0.025s sys 0.014s
I20240506 15:14:13.050402 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.052173 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:13.049147 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.053381 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:13.053781 21073 raft_consensus.cc:348] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.054425 21073 raft_consensus.cc:374] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.054663 21073 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.056918 20950 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.057545 20956 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.060369 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.060242 21072 raft_consensus.cc:348] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.061125 21072 raft_consensus.cc:374] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.061408 21072 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.061743 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.062156 21504 consensus_queue.cc:1046] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:13.063618 20845 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.065106 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:13.064903 21072 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.068445 21073 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.068396 20779 catalog_manager.cc:5484] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:13.070855 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.072122 21504 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.073640 21504 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.073802 21388 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:13.074206 21388 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:13.074569 21388 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.073093 20846 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.076107 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.073668 21506 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.077240 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:13.079145 20846 leader_election.cc:341] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.079802 21448 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.080860 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.081100 21448 consensus_queue.cc:1046] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:13.081845 21028 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.082031 21506 raft_consensus.cc:3032] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:13.082427 21506 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.074499 21504 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.083428 21027 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:13.083179 21506 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.083976 21504 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.084569 21028 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.085963 20845 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.062397 21023 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:13.086975 21436 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.090221 21032 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.091360 21436 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.093065 21074 ts_tablet_manager.cc:1398] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.074s user 0.008s sys 0.000s
I20240506 15:14:13.094475 21074 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
W20240506 15:14:13.119591 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70967.8us
I20240506 15:14:13.120348 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.121484 21496 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.085251 21526 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:13.133484 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47576.9us
I20240506 15:14:13.130293 21073 ts_tablet_manager.cc:1398] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.110s user 0.007s sys 0.002s
W20240506 15:14:13.135030 20845 leader_election.cc:341] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.136123 20845 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.136934 21079 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.137236 20845 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.139793 20998 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.140895 21079 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.141300 20845 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.142069 21079 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.142419 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.144380 20998 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.144877 21072 ts_tablet_manager.cc:1398] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.094s user 0.006s sys 0.001s
I20240506 15:14:13.148865 20924 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.153374 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:13.146487 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 56314.2us
I20240506 15:14:13.154875 20845 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.150188 21031 raft_consensus.cc:1226] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.157647 21496 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.157948 21340 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.158215 21073 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:13.135645 21526 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:13.133672 21524 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.147233 21448 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.153544 21493 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.160240 21456 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.158540 21340 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.156688 21421 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:13.174373 21421 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.160629 21526 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.201834 21536 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:13.208424 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 53275.8us
I20240506 15:14:13.225700 21074 tablet_bootstrap.cc:654] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
W20240506 15:14:13.226713 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 73555.5us
I20240506 15:14:13.175714 21421 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:13.227988 20921 leader_election.cc:341] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:13.233776 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 79002.3us
I20240506 15:14:13.234570 20845 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:13.235082 20921 leader_election.cc:341] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.235651 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.235962 21484 consensus_queue.cc:1046] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:13.236721 20845 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.237128 21484 raft_consensus.cc:3032] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.225567 21506 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:13.236290 20921 leader_election.cc:341] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.238332 21484 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.239457 21388 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.246424 21028 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:13.246513 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.248147 21484 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.249238 21484 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.249433 21536 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.251133 21448 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.251747 21448 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.253679 21072 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:13.253531 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:13.254348 21525 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:13.242828 21024 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.235747 21340 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:13.256484 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:13.257159 21446 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:13.257807 21444 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:13.258946 21444 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:13.259322 21444 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.267028 21388 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:13.267462 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 57429.2us
I20240506 15:14:13.269661 21446 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.270591 21446 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:13.272104 20846 leader_election.cc:341] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.273085 21446 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.277237 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.289997 21079 consensus_queue.cc:1046] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:13.291020 21079 consensus_queue.cc:1225] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer ecededcd854842d4867fda9fa63a4b66 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240506 15:14:13.291429 21539 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.291568 21079 raft_consensus.cc:3032] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.291431 20845 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.292305 21079 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.293608 21506 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.293254 21079 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:13.328912 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 74085.8us
I20240506 15:14:13.293435 20845 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:13.342217 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 47633.4us
I20240506 15:14:13.351087 21072 tablet_bootstrap.cc:654] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:13.352633 21073 tablet_bootstrap.cc:654] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:13.356813 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:13.357646 20954 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.357791 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.358816 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.359645 20880 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:13.359813 21074 tablet_bootstrap.cc:492] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:13.344729 21525 consensus_queue.cc:1046] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240506 15:14:13.360304 21074 ts_tablet_manager.cc:1367] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.266s user 0.026s sys 0.011s
I20240506 15:14:13.360683 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.361343 20879 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:13.361687 21079 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.356881 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.362073 21430 raft_consensus.cc:3032] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.362445 21430 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.363219 21430 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.346859 21446 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.364363 21430 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:13.369022 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 75693.1us
I20240506 15:14:13.371068 20845 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.360718 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.378278 20879 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240506 15:14:13.378667 20879 pending_rounds.cc:76] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 0
I20240506 15:14:13.382616 20845 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.371492 21388 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.378894 20879 pending_rounds.cc:97] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240506 15:14:13.389813 20879 raft_consensus.cc:2866] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:13.390681 21541 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.391446 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.389150 20845 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.397785 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.388860 21074 raft_consensus.cc:348] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.398342 21074 raft_consensus.cc:374] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.398619 21074 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.400097 21074 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.384992 21484 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20240506 15:14:13.403158 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.403857 21484 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.404227 21484 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.404410 21543 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.392374 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.405023 21484 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.406225 21484 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:13.370733 21525 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240506 15:14:13.392649 21030 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:13.399601 20845 leader_election.cc:341] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.391808 21444 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.415580 20845 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.414489 21479 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.418435 21479 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.419137 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.419239 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.420010 20880 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.420660 21479 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.409333 21388 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.429426 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.430159 21028 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:13.430936 21444 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:13.433774 20845 leader_election.cc:341] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.434104 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:13.437711 21547 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:13.438127 21547 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:13.438493 21547 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.434263 20845 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.392836 21544 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.442528 21541 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.443830 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.446378 21543 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.444021 21074 ts_tablet_manager.cc:1398] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.083s user 0.004s sys 0.002s
I20240506 15:14:13.446046 20845 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.448499 20845 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.449996 20845 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.451848 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.450675 21544 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.453095 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.448858 21543 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.455089 21544 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.453941 21541 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:13.456952 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.448504 20924 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.424871 20878 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:13.460808 21541 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.464798 20880 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:13.465472 21544 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.476974 21544 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.477789 20954 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:13.479434 21544 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.480733 21028 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:13.451153 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:13.483374 20845 leader_election.cc:341] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.483801 20845 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.472334 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.475140 21525 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.453984 20882 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:13.487509 21074 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Bootstrap starting.
I20240506 15:14:13.485476 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.485889 21421 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240506 15:14:13.488456 21525 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:13.485911 21388 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.487217 21539 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.501963 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.486557 21542 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.510048 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.498940 21525 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:13.516512 21525 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.514482 21539 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.514819 21479 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.512535 21526 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.496793 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.520411 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:13.525740 21073 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:13.526189 21073 ts_tablet_manager.cc:1367] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.368s user 0.031s sys 0.038s
I20240506 15:14:13.505507 21444 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240506 15:14:13.539551 20924 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.537881 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:13.543561 21446 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:13.559664 20845 leader_election.cc:341] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.559770 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.565124 21526 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.566779 21340 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.569962 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:13.589788 21074 tablet_bootstrap.cc:654] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:13.597362 21073 raft_consensus.cc:348] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.598304 21073 raft_consensus.cc:374] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.598882 21073 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.596593 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.599561 21073 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.602412 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.603897 21444 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:13.603564 21525 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.604586 21444 raft_consensus.cc:686] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.605397 21444 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.610954 21072 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:13.611552 21072 ts_tablet_manager.cc:1367] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.358s user 0.008s sys 0.011s
I20240506 15:14:13.612387 21547 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.615190 21547 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.615057 20777 catalog_manager.cc:5484] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:13.618525 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:13.619230 21030 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.624830 21525 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.625119 21073 ts_tablet_manager.cc:1398] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.099s user 0.007s sys 0.001s
I20240506 15:14:13.628124 21073 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:13.614449 21072 raft_consensus.cc:348] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.633803 21072 raft_consensus.cc:374] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.634192 21072 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.635383 21072 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.642035 21073 tablet_bootstrap.cc:654] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:13.643675 21072 ts_tablet_manager.cc:1398] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20240506 15:14:13.646126 21074 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: No bootstrap required, opened a new log
I20240506 15:14:13.646459 21072 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:13.646773 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:13.646781 21074 ts_tablet_manager.cc:1367] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Time spent bootstrapping tablet: real 0.176s user 0.017s sys 0.009s
W20240506 15:14:13.648942 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 51775.3us
W20240506 15:14:13.653532 20845 leader_election.cc:341] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:13.656638 20921 leader_election.cc:341] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.655284 21074 raft_consensus.cc:348] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.658412 21074 raft_consensus.cc:374] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.658720 21074 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.659385 21074 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.659911 21072 tablet_bootstrap.cc:654] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:13.663930 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.665154 21074 ts_tablet_manager.cc:1398] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240506 15:14:13.650310 21030 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:13.666780 21444 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:13.667378 21444 raft_consensus.cc:686] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.668519 21444 consensus_queue.cc:237] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.674005 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:13.675499 20924 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.676904 21525 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.680133 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.694509 21544 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:13.724380 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.708719 21430 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:13.725137 20952 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:13.727816 20878 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:13.730114 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.730798 20881 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.731025 21544 raft_consensus.cc:686] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.732360 21544 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.733788 20877 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:13.735391 21430 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.736714 21430 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.737347 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.738255 20874 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.763641 21073 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:13.764187 21073 ts_tablet_manager.cc:1367] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.013s
I20240506 15:14:13.796819 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.821645 21536 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:13.822132 21536 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:13.822494 21536 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:13.837159 21444 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:13.837937 20956 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.919317 21448 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.921628 20881 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:13.921414 21448 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.933394 21073 raft_consensus.cc:348] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.935314 21073 raft_consensus.cc:374] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:13.935672 21073 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:13.936656 21073 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.946025 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:13.947031 21484 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:13.947661 21484 raft_consensus.cc:686] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:13.951892 21448 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.932152 21539 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.953524 21539 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.841936 21543 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.958912 21446 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:13.959285 21539 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:13.959846 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.959981 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.938912 21535 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20240506 15:14:13.961222 20845 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.961611 20950 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:13.962973 21535 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:13.963498 21535 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:13.962047 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.964206 21484 consensus_queue.cc:237] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.962283 21073 ts_tablet_manager.cc:1398] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.198s user 0.009s sys 0.013s
I20240506 15:14:13.865782 21450 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:13.965546 20845 leader_election.cc:341] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.969393 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:13.971158 21569 log.cc:924] Time spent T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20240506 15:14:13.969475 21535 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.974738 21535 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:13.974969 20921 consensus_peers.cc:485] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20240506 15:14:13.977460 20846 leader_election.cc:341] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.977893 20846 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:13.839577 21444 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.126s
I20240506 15:14:13.981669 21340 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:13.981881 21506 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:13.982185 21340 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.976969 21450 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:13.986737 21340 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:13.988870 21073 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:13.839327 21542 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20240506 15:14:13.993774 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:13.995782 21072 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:13.991947 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:13.993011 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:13.997301 20996 leader_election.cc:341] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:13.997720 21072 ts_tablet_manager.cc:1367] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.352s user 0.020s sys 0.008s
I20240506 15:14:13.980508 21539 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240506 15:14:14.005496 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.000231 21072 raft_consensus.cc:348] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.006568 21072 raft_consensus.cc:374] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.006965 21072 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.008659 21072 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.001108 21544 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:14.022131 21544 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:14.025851 21544 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.012709 20954 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 0.
I20240506 15:14:14.023535 21542 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:13.992395 20874 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:14.035271 21450 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.038264 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:13.972687 21543 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:14.042793 20921 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.044957 21072 ts_tablet_manager.cc:1398] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20240506 15:14:14.045025 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.046164 21072 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.046633 20879 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:14.051090 20881 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:14.051374 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.053143 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.053977 21493 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.054350 21493 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.054705 21493 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.056491 21543 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.056769 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.060011 21536 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.063041 21536 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.063306 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.063719 21279 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.064280 21279 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:14.064930 20845 leader_election.cc:341] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.066805 21279 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.060353 21544 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:14.067876 20779 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.074910 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.079319 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.079778 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.080303 20954 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:14.080401 21536 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.080837 21536 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.081200 21536 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.079592 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.082748 20957 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.085762 21450 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.086413 21450 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.093914 21029 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.112429 21029 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:14.087893 21450 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.117473 20957 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
W20240506 15:14:14.133327 20846 leader_election.cc:341] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.134745 21526 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:14.135757 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.127700 21493 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.140053 21448 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:14.152657 20876 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:14.129247 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.155462 21073 tablet_bootstrap.cc:654] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.154779 21448 raft_consensus.cc:686] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:14.159070 20998 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.165405 21072 tablet_bootstrap.cc:654] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.168030 21031 raft_consensus.cc:2379] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:14.166119 21444 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240506 15:14:14.157933 21448 consensus_queue.cc:237] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.167522 21540 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.180706 21446 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.182644 21446 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.186821 21446 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.193250 20845 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.194546 21543 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.196856 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.197679 20876 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:14.197394 20779 catalog_manager.cc:5484] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:14.199018 20779 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 2 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.200726 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.204821 21493 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.207427 20779 catalog_manager.cc:5484] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.207614 21446 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 15:14:14.187502 21536 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.209569 20779 catalog_manager.cc:5484] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.212378 20779 catalog_manager.cc:5484] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:14.215052 20779 catalog_manager.cc:5484] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.205796 21484 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.220584 21484 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.220958 21484 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.219951 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.221068 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.219951 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.223479 21536 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.224642 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:14.230609 20921 leader_election.cc:341] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:14.234804 20996 leader_election.cc:341] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.234728 20779 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
W20240506 15:14:14.240749 20845 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.249425 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.226253 21450 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.260577 20957 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:14.264206 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.266163 21073 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:14.266886 21073 ts_tablet_manager.cc:1367] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.286s user 0.017s sys 0.012s
I20240506 15:14:14.268523 21536 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.304152 21450 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.309406 21543 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 15:14:14.313503 20921 consensus_peers.cc:485] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:14.315629 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.321720 20878 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:14.337910 21072 tablet_bootstrap.cc:492] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.338407 21072 ts_tablet_manager.cc:1367] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.292s user 0.030s sys 0.037s
I20240506 15:14:14.332630 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.348853 21073 raft_consensus.cc:348] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.349635 21073 raft_consensus.cc:374] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.349895 21073 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.350634 21073 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.345297 21444 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240506 15:14:14.356513 21525 consensus_queue.cc:1035] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:14.359094 21073 ts_tablet_manager.cc:1398] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.092s user 0.004s sys 0.002s
I20240506 15:14:14.360139 21073 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
W20240506 15:14:14.360756 20921 leader_election.cc:341] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.361346 20921 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.362154 21526 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.363569 21072 raft_consensus.cc:348] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.364549 21072 raft_consensus.cc:374] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.364835 21072 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.366312 21072 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.368567 21072 ts_tablet_manager.cc:1398] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.028s user 0.004s sys 0.003s
I20240506 15:14:14.369496 21072 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.372539 21484 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.336114 21543 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20240506 15:14:14.381978 20876 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:14.386337 21072 tablet_bootstrap.cc:654] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.386755 21540 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.387954 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.390170 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.396095 20876 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.396389 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:14.398479 20921 leader_election.cc:341] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
W20240506 15:14:14.401712 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:14.404170 21340 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:14.404762 21340 raft_consensus.cc:686] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:14.405613 21340 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.406833 21073 tablet_bootstrap.cc:654] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.414037 21544 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:14.390679 21484 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.433769 20876 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:14.447589 21072 tablet_bootstrap.cc:492] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.448128 21072 ts_tablet_manager.cc:1367] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.079s user 0.023s sys 0.018s
I20240506 15:14:14.450450 21072 raft_consensus.cc:348] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.450980 21072 raft_consensus.cc:374] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.451192 21072 raft_consensus.cc:729] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.451669 21072 consensus_queue.cc:260] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.457667 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.458770 21484 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:14.459435 21484 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:14.460292 21484 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.480749 21279 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.481276 21279 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.484355 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.485164 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.489756 20998 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.490681 21279 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.491276 21340 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.499269 20777 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:14.505970 21484 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.506516 21484 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.508792 21484 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.513837 21073 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:14.516796 21073 ts_tablet_manager.cc:1367] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.157s user 0.033s sys 0.037s
I20240506 15:14:14.513969 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.517390 21072 ts_tablet_manager.cc:1398] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.069s user 0.001s sys 0.003s
I20240506 15:14:14.517895 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:14.519290 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.518481 21072 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.520413 21484 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:14.520906 21484 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.520722 21073 raft_consensus.cc:348] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.521526 21073 raft_consensus.cc:374] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.521392 21484 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.521911 21073 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.522650 21073 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.526791 21073 ts_tablet_manager.cc:1398] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240506 15:14:14.527805 21073 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Bootstrap starting.
I20240506 15:14:14.534780 21073 tablet_bootstrap.cc:654] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.535683 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.535187 20779 catalog_manager.cc:5484] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.539222 21484 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.545092 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.545830 21030 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.546717 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.563598 21030 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:14.565466 21072 tablet_bootstrap.cc:654] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.564319 21484 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.566633 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:14.568833 21444 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:14.569671 21444 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:14.569818 21073 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: No bootstrap required, opened a new log
I20240506 15:14:14.571235 21073 ts_tablet_manager.cc:1367] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.008s
I20240506 15:14:14.573486 21526 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:14.573585 20780 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.570685 21444 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.574443 21073 raft_consensus.cc:348] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.578178 21073 raft_consensus.cc:374] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.578529 21073 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.579241 21073 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.582993 21073 ts_tablet_manager.cc:1398] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240506 15:14:14.590770 21444 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.591532 21444 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.595007 21444 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.597769 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.598738 20876 raft_consensus.cc:2379] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 1: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:14.600561 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.603664 20921 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.605063 21484 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.617447 21484 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.623167 21030 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.628443 21493 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:14.629209 21493 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.631593 20780 catalog_manager.cc:5484] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.636372 21493 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.637332 21444 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.637835 21444 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.638195 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.639142 20874 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:14.639791 21444 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.642673 20998 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.646693 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.647611 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:14.644536 21340 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:14.646603 21279 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.652516 21526 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:14.655719 21072 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.652515 21340 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:14.653316 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.656230 21340 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:14.656831 20957 raft_consensus.cc:2379] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:14.655759 21279 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.653779 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.658607 21279 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.658891 21526 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:14.656358 21072 ts_tablet_manager.cc:1367] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.138s user 0.013s sys 0.022s
I20240506 15:14:14.661443 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.660156 21526 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.660806 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.662508 21526 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.663234 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.665623 21072 raft_consensus.cc:348] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.666371 21072 raft_consensus.cc:374] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.666643 21072 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.667259 21072 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.669605 21072 ts_tablet_manager.cc:1398] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240506 15:14:14.670565 21072 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.670466 20996 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.675429 21340 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.679160 21340 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.679524 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.680794 21526 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.681032 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.682258 20954 raft_consensus.cc:2379] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:14.682936 21526 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.684376 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.685200 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.686839 21031 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.697413 21523 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:14.697968 21072 tablet_bootstrap.cc:654] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.703536 21027 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.703878 21448 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:14.710361 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:14.712862 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:14.713691 21484 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:14.714668 21484 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:14.716056 21484 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.724954 21450 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.725484 21450 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.727459 21450 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.717182 20874 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:14.729916 20998 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.730079 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.731159 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.731884 20881 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 0.
I20240506 15:14:14.732043 21523 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:14.732760 21523 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:14.733875 21523 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.737948 20924 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.743417 21072 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.743952 21072 ts_tablet_manager.cc:1367] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20240506 15:14:14.746421 21072 raft_consensus.cc:348] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.747164 21072 raft_consensus.cc:374] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.747465 21072 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
W20240506 15:14:14.747798 20921 leader_election.cc:341] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:14.748302 21072 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.749989 21072 ts_tablet_manager.cc:1398] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240506 15:14:14.750946 21072 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.756795 21072 tablet_bootstrap.cc:654] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.759163 21444 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.759635 21444 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.760003 21444 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
W20240506 15:14:14.761472 20845 consensus_peers.cc:485] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Couldn't send request to peer 7610624a16b44896b6aacea1928ab20e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:14.772166 21444 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.780756 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.781476 20874 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.790551 21072 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.791121 21072 ts_tablet_manager.cc:1367] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.006s
I20240506 15:14:14.793804 21072 raft_consensus.cc:348] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.794615 21072 raft_consensus.cc:374] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.794929 21072 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.795622 21072 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.797353 21072 ts_tablet_manager.cc:1398] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240506 15:14:14.798457 21072 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.804523 21072 tablet_bootstrap.cc:654] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.808728 20874 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:14.810612 20924 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.813278 21450 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:14.802681 21444 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.813980 21450 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:14.814823 21450 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.816677 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.817260 21031 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.824084 21031 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:14.824851 21523 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.816076 20779 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 2 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:14.825490 21523 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.828701 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.829432 20881 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:14.830394 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.831059 20957 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 0.
I20240506 15:14:14.832450 20998 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.833079 21340 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 15:14:14.833433 21340 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.833746 21340 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.828701 21523 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.847591 21340 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.852377 21072 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.852998 21072 ts_tablet_manager.cc:1367] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.019s
I20240506 15:14:14.852087 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.853500 20881 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.855530 21072 raft_consensus.cc:348] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.856372 21072 raft_consensus.cc:374] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.855748 20777 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.856654 21072 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.857381 21072 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.859405 21340 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.859519 21072 ts_tablet_manager.cc:1398] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240506 15:14:14.860464 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.861018 20957 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.865425 20881 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:14.866415 20998 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.867228 21523 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader election won for term 1
I20240506 15:14:14.867305 20957 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:14.857383 20777 catalog_manager.cc:5484] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 1 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:14.869642 21523 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:14.870509 21523 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.873106 21072 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.879994 21072 tablet_bootstrap.cc:654] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.886711 21523 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:14.889993 21072 tablet_bootstrap.cc:492] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.890836 21072 ts_tablet_manager.cc:1367] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240506 15:14:14.891173 20777 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:14.893210 21072 raft_consensus.cc:348] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.893761 21072 raft_consensus.cc:374] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.893965 21072 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.894507 21072 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.897212 21072 ts_tablet_manager.cc:1398] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240506 15:14:14.898823 21072 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: Bootstrap starting.
I20240506 15:14:14.905575 21072 tablet_bootstrap.cc:654] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: Neither blocks nor log segments found. Creating new log.
I20240506 15:14:14.911767 21535 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.912664 21535 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.915784 21535 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.916484 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.917148 20957 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:14.919051 20846 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:14.919229 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.919675 21535 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:14.920009 21072 tablet_bootstrap.cc:492] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: No bootstrap required, opened a new log
I20240506 15:14:14.920188 21535 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.920766 21535 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:14.920042 21031 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:14.922439 21072 ts_tablet_manager.cc:1367] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.002s
I20240506 15:14:14.924432 21072 raft_consensus.cc:348] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.924939 21072 raft_consensus.cc:374] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 15:14:14.925102 21072 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Initialized, Role: FOLLOWER
I20240506 15:14:14.925506 21072 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.925906 21448 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.926216 21279 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.926436 21448 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.926728 21279 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.927444 21072 ts_tablet_manager.cc:1398] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 15:14:14.928474 21448 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.928858 21279 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.930044 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.930974 20957 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:14.931017 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.931984 21029 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.935353 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.936295 20878 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 1: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:14.930102 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.939229 21031 raft_consensus.cc:2354] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.937541 20998 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:14.937629 21448 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.940061 21448 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.940727 20845 leader_election.cc:398] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.936786 21523 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240506 15:14:14.941184 20845 leader_election.cc:401] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:14.942070 21523 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:14.943073 21543 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.944128 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.953249 20952 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:14.944423 21279 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.954542 21279 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.955180 21544 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.955749 21544 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.955767 21535 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.945222 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.957247 21031 raft_consensus.cc:2379] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:14.957504 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.957942 21544 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.958163 20878 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:14.958354 21535 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.959753 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:14.960317 20998 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.960222 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.961215 21493 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:14.961277 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:14.961706 21493 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.962034 21031 raft_consensus.cc:2354] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.962108 21493 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.963410 20845 leader_election.cc:398] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.963689 20845 leader_election.cc:401] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:14.963802 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.964273 21536 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:14.964579 21544 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.964658 21536 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:14.942301 21448 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.965179 21536 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:14.963819 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.968322 21279 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.952117 21543 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.970427 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:14.974548 21536 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.976297 20957 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:14.977512 21340 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.978080 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:14.977977 21340 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.978770 21030 raft_consensus.cc:2379] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:14.978910 21493 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.980165 21027 raft_consensus.cc:2354] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.981067 21493 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.957484 21523 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 15:14:14.983999 21536 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:14.981781 21544 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.985890 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:14.986519 20878 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.981978 21340 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:14.986596 20845 leader_election.cc:398] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.987437 20845 leader_election.cc:401] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:14.988548 21340 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:14.988981 21340 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:14.991757 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:14.992090 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.992595 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:14.992796 20876 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:14.993216 20874 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:14.994046 20998 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.995402 20878 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:14.996290 21523 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:14.996781 21448 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:14.997125 21523 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.997537 21523 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:14.997642 20998 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:14.998772 21279 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:14.999300 21279 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:14.999673 21279 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:14.998927 20998 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:15.005029 20957 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:15.005664 21523 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.007210 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:15.007439 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:15.006273 21430 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.008530 21340 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.008354 21525 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.006731 21493 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:15.007829 20954 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:15.011160 21523 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.012487 21493 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:15.012720 21279 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.013772 21493 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.015149 21279 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.021495 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:15.016755 20954 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:15.018465 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:15.022750 20956 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:15.024816 21544 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:15.025399 21544 raft_consensus.cc:686] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:15.026296 21544 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.018638 21027 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240506 15:14:15.019412 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:15.020077 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:15.034226 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:15.034649 21430 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.035609 21525 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:15.035979 21525 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:15.036782 21525 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.037176 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:15.037783 21525 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:15.044282 20998 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:15.044685 20777 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:15.045439 21493 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:15.047895 20956 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:15.049381 20996 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:15.050266 21279 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:15.050815 21279 raft_consensus.cc:686] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:15.051771 21279 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.051774 20779 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 2, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 2 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:15.064967 21279 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:15.065490 21279 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.069001 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:15.069353 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:15.069758 20878 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:15.070128 20956 raft_consensus.cc:2379] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:15.070817 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:15.068356 21279 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.071780 21493 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:15.072191 21493 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:15.072602 21493 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:15.080053 21493 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.089192 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:15.090770 20956 raft_consensus.cc:2379] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:15.091434 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:15.088989 21493 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.089960 20779 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:15.101689 21027 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.105181 21444 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.109047 20878 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:15.110620 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:15.111757 21279 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:15.112231 21279 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:15.113090 21279 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.119220 20878 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.121062 21430 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.146605 21027 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:15.150935 20779 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 2 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:15.234709 21444 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.245903 21484 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.250844 20874 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.252033 21484 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.270073 21031 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.272586 21430 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:15.303380 21493 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.307013 20954 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:15.308444 21493 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.320152 21540 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:15.332163 20874 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:15.333838 21279 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.365746 21540 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:15.366250 21540 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.368532 21540 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:15.371215 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:15.374796 21031 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 0 FOLLOWER]: Advancing to term 1
I20240506 15:14:15.381158 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:15.382719 20996 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:15.392539 21540 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 15:14:15.436558 20828 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240506 15:14:15.493079 20881 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.494967 21279 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.495117 20954 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.496682 21540 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.541291 20954 raft_consensus.cc:1226] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:15.542623 21544 consensus_queue.cc:1046] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.543380 21544 raft_consensus.cc:3032] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:15.543715 21544 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:15.544631 21544 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.545809 21544 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:15.551824 20881 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240506 15:14:15.552081 20881 pending_rounds.cc:76] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 1
I20240506 15:14:15.552320 20881 pending_rounds.cc:97] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240506 15:14:15.552603 20881 raft_consensus.cc:2866] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:15.554865 20874 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:15.555508 21279 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.556324 21540 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.565244 20954 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:15.567500 21540 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:15.578234 20956 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:15.580657 21493 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:15.774134 21544 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:15.774919 21544 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.777252 21544 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:15.777654 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:15.778331 20957 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:15.785096 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:15.785812 21031 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:15.785439 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:15.787009 21543 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:15.787357 21543 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:15.787636 21543 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:15.794117 21543 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.796397 21543 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:15.796939 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:15.797412 21031 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:15.797782 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:15.798357 20956 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:15.803918 21031 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:15.804452 20956 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:15.805361 20845 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:15.805948 21543 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:15.806432 21543 raft_consensus.cc:686] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:15.807250 21543 consensus_queue.cc:237] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:15.818639 20779 catalog_manager.cc:5484] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 0 to 2, leader changed from <none> to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 2 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:16.211828 21525 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:16.212345 21525 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:16.215092 21525 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:16.215709 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:16.217670 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:16.218556 21030 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:16.219882 20921 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:16.220518 21525 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:16.220903 21525 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:16.221166 21525 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:16.228802 21525 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:16.230548 21525 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:16.231555 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:16.231467 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:16.232368 21030 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:16.237289 21030 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:16.238359 20921 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:16.239055 21525 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:16.239442 21525 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:16.240159 21525 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:16.251698 20779 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:16.294736 21030 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:16.296116 21544 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:16.311758 20957 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 15:14:16.313148 21543 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:16.529485 21543 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:16.530190 21543 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:16.532007 21543 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:16.532881 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:16.533304 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:16.534003 21031 raft_consensus.cc:2354] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:16.534847 20845 leader_election.cc:398] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:16.535046 20845 leader_election.cc:401] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:16.535502 21544 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:16.538986 19726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 15:14:16.543143 21544 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
W20240506 15:14:16.567205 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 15:14:16.568481 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:16.570694 19726 server_base.cc:1034] running on GCE node
W20240506 15:14:16.571657 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 15:14:16.577131 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240506 15:14:16.577594 19726 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240506 15:14:16.577894 19726 hybrid_clock.cc:648] HybridClock initialized: now 1715008456577865 us; error 0 us; skew 500 ppm
I20240506 15:14:16.579011 19726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 15:14:16.586472 19726 webserver.cc:460] Webserver started at http://127.19.67.132:33375/ using document root <none> and password file <none>
I20240506 15:14:16.587021 19726 fs_manager.cc:359] Metadata directory not provided
I20240506 15:14:16.587219 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 15:14:16.587522 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 15:14:16.589017 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-3-root/instance:
uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
format_stamp: "Formatted at 2024-05-06 15:14:16 on dist-test-slave-8gjm"
I20240506 15:14:16.594972 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240506 15:14:16.599807 21652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:16.600733 19726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 15:14:16.601163 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-3-root
uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
format_stamp: "Formatted at 2024-05-06 15:14:16 on dist-test-slave-8gjm"
I20240506 15:14:16.601454 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 15:14:16.621901 19726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 15:14:16.624244 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 15:14:16.627274 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 15:14:16.630380 19726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 15:14:16.630595 19726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:16.630811 19726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 15:14:16.630954 19726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 15:14:16.715642 21027 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:16.719017 21430 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 15:14:16.746229 20874 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:16.747905 21525 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 15:14:16.793779 19726 rpc_server.cc:307] RPC server started. Bound to: 127.19.67.132:34421
I20240506 15:14:16.793964 21718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.67.132:34421 every 8 connection(s)
I20240506 15:14:16.800367 19726 auto_leader_rebalancer-test.cc:405] add a tserver: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:16.818593 21719 heartbeater.cc:347] Connected to a master server at 127.19.67.190:37083
I20240506 15:14:16.819095 21719 heartbeater.cc:464] Registering TS with master...
I20240506 15:14:16.819936 21719 heartbeater.cc:511] Master 127.19.67.190:37083 requested a full tablet report, sending...
I20240506 15:14:16.821982 20780 ts_manager.cc:194] Registered new tserver with Master: 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:16.823638 20780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39518
I20240506 15:14:17.577454 21027 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:17.584861 20874 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:17.585705 20956 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:17.586345 21726 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:17.588400 21726 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240506 15:14:17.599418 20996 consensus_peers.cc:485] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc90d9cec20d4d219e474797b7115172. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:17.600764 21712 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.605183 20874 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.609153 20956 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.630887 20780 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.644979 20874 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:17.651890 21027 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:17.653523 21543 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:17.653959 20956 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:17.656571 21536 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:17.671711 21543 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.673256 20957 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.681900 21031 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
W20240506 15:14:17.682971 20846 consensus_peers.cc:485] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc05b4289a334b34a5f571e608bc9191. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:17.687642 21027 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:17.696149 20777 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:17.704604 20874 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:17.705096 20957 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:17.706035 21712 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240506 15:14:17.708087 20996 consensus_peers.cc:485] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f7a6eab190f4de2bbd9e6b2ff60cdc1. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:17.709666 21712 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:17.721124 21493 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.722846 20956 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.724586 20874 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.735622 21027 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:17.739317 20779 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.766654 20956 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240506 15:14:17.767220 20996 consensus_peers.cc:485] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9178d1bf59624e7f8948c33552e74a00. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:17.768056 21493 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:17.771311 20874 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:17.776397 21729 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:17.789021 21726 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.792773 20956 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.805505 20881 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.811220 20780 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:17.825947 21719 heartbeater.cc:503] Master 127.19.67.190:37083 was elected leader, sending a full tablet report...
I20240506 15:14:18.168166 21747 ts_tablet_manager.cc:902] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:18.174963 21747 tablet_copy_client.cc:316] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:18.221920 21752 ts_tablet_manager.cc:902] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:18.230175 21752 tablet_copy_client.cc:316] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.129:44145
I20240506 15:14:18.236493 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet cc90d9cec20d4d219e474797b7115172 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:18.237269 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet cc90d9cec20d4d219e474797b7115172 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-cc90d9cec20d4d219e474797b7115172
I20240506 15:14:18.246511 21042 tablet_copy_source_session.cc:205] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:18.256160 21747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc90d9cec20d4d219e474797b7115172. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 15:14:18.278506 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 47333us
I20240506 15:14:18.280136 21747 tablet_copy_client.cc:791] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:18.280817 21747 tablet_copy_client.cc:655] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:18.358541 20893 tablet_copy_service.cc:140] P 00fe71c1675441379f5f2c7490f9e63a: Received BeginTabletCopySession request for tablet bc05b4289a334b34a5f571e608bc9191 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49818)
I20240506 15:14:18.359169 20893 tablet_copy_service.cc:161] P 00fe71c1675441379f5f2c7490f9e63a: Beginning new tablet copy session on tablet bc05b4289a334b34a5f571e608bc9191 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49818: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-bc05b4289a334b34a5f571e608bc9191
I20240506 15:14:18.370916 20893 tablet_copy_source_session.cc:205] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:18.376047 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc05b4289a334b34a5f571e608bc9191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:18.397521 21747 tablet_copy_client.cc:531] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:18.410713 21761 ts_tablet_manager.cc:902] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:18.420542 21761 tablet_copy_client.cc:316] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:18.422317 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet 9178d1bf59624e7f8948c33552e74a00 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:18.422914 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet 9178d1bf59624e7f8948c33552e74a00 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-9178d1bf59624e7f8948c33552e74a00
I20240506 15:14:18.436615 21752 tablet_copy_client.cc:791] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:18.437240 21752 tablet_copy_client.cc:655] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:18.440099 21762 ts_tablet_manager.cc:902] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:18.441857 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 59000us
I20240506 15:14:18.447710 21747 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:18.452172 21762 tablet_copy_client.cc:316] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:18.459352 21042 tablet_copy_source_session.cc:205] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:18.461247 21752 tablet_copy_client.cc:531] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:18.470404 21761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9178d1bf59624e7f8948c33552e74a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:18.473461 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:18.474231 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-5f7a6eab190f4de2bbd9e6b2ff60cdc1
I20240506 15:14:18.483150 21752 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:18.496307 21761 tablet_copy_client.cc:791] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:18.500102 21042 tablet_copy_source_session.cc:205] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:18.505312 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:18.505368 21761 tablet_copy_client.cc:655] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:18.542342 21761 tablet_copy_client.cc:531] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:18.563676 21761 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:18.566392 21762 tablet_copy_client.cc:791] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:18.567092 21762 tablet_copy_client.cc:655] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:18.580210 21747 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:18.580803 21762 tablet_copy_client.cc:531] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:18.586795 21747 tablet_bootstrap.cc:492] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:18.587397 21747 ts_tablet_manager.cc:1367] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.140s user 0.038s sys 0.014s
I20240506 15:14:18.594843 21762 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:18.598317 21747 raft_consensus.cc:348] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.599238 21747 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:18.604460 21747 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.624502 21752 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:18.624761 21761 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:18.625451 21752 tablet_bootstrap.cc:492] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:18.638911 21762 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:18.640695 21761 tablet_bootstrap.cc:492] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:18.642971 21752 ts_tablet_manager.cc:1367] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.160s user 0.024s sys 0.018s
I20240506 15:14:18.643358 21761 ts_tablet_manager.cc:1367] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.015s
I20240506 15:14:18.643460 21762 tablet_bootstrap.cc:492] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:18.642104 21747 ts_tablet_manager.cc:1398] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.047s user 0.013s sys 0.005s
I20240506 15:14:18.645191 21762 ts_tablet_manager.cc:1367] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.051s user 0.022s sys 0.004s
I20240506 15:14:18.645967 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-cc90d9cec20d4d219e474797b7115172 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:18.646420 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-cc90d9cec20d4d219e474797b7115172 on tablet cc90d9cec20d4d219e474797b7115172 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:18.648339 21752 raft_consensus.cc:348] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.649652 21752 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:18.650089 21761 raft_consensus.cc:348] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.650334 21752 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.651211 21762 raft_consensus.cc:348] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.652077 21762 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:18.653030 21752 ts_tablet_manager.cc:1398] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240506 15:14:18.652758 21762 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.654911 21762 ts_tablet_manager.cc:1398] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240506 15:14:18.655411 20893 tablet_copy_service.cc:342] P 00fe71c1675441379f5f2c7490f9e63a: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-bc05b4289a334b34a5f571e608bc9191 received from {username='slave'} at 127.0.0.1:49818
I20240506 15:14:18.655799 20893 tablet_copy_service.cc:433] P 00fe71c1675441379f5f2c7490f9e63a: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-bc05b4289a334b34a5f571e608bc9191 on tablet bc05b4289a334b34a5f571e608bc9191 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:18.657305 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-5f7a6eab190f4de2bbd9e6b2ff60cdc1 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:18.657691 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-5f7a6eab190f4de2bbd9e6b2ff60cdc1 on tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:18.650933 21761 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:18.663480 21761 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:18.665920 21761 ts_tablet_manager.cc:1398] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240506 15:14:18.667800 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-9178d1bf59624e7f8948c33552e74a00 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:18.668215 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-9178d1bf59624e7f8948c33552e74a00 on tablet 9178d1bf59624e7f8948c33552e74a00 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:18.675822 21689 raft_consensus.cc:1203] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:18.718039 21689 raft_consensus.cc:1203] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:18.894560 21689 raft_consensus.cc:1203] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:14:18.929201 21689 raft_consensus.cc:1203] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:14:19.273087 21536 raft_consensus.cc:1050] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:19.275986 21536 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.289592 21689 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.290815 21536 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:19.292695 21030 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.292850 20950 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.294509 21536 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:19.295557 21543 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:19.299985 21764 raft_consensus.cc:1050] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:19.305742 21764 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:19.310122 20811 debug-util.cc:398] Leaking SignalData structure 0x7b080003abe0 after lost signal to thread 19729
W20240506 15:14:19.311206 20811 debug-util.cc:398] Leaking SignalData structure 0x7b080003d280 after lost signal to thread 20814
W20240506 15:14:19.313216 20811 debug-util.cc:398] Leaking SignalData structure 0x7b080010f4e0 after lost signal to thread 20982
W20240506 15:14:19.314812 20811 debug-util.cc:398] Leaking SignalData structure 0x7b08003072a0 after lost signal to thread 21056
W20240506 15:14:19.316313 20811 debug-util.cc:398] Leaking SignalData structure 0x7b0800307c60 after lost signal to thread 21718
I20240506 15:14:19.322214 21543 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.330253 21690 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.335115 20956 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.337666 21030 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.351583 20780 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.367246 21689 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.368059 20876 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.369452 21760 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:19.370350 21726 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:19.379067 20956 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:19.389976 21726 raft_consensus.cc:1050] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:19.393553 21726 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.399811 21729 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:19.421448 21779 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.424937 20950 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.432658 21690 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.447129 21688 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.448937 21779 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.453136 20950 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.455130 21729 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.460489 21726 raft_consensus.cc:1050] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:19.462761 21726 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.485641 20878 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.494838 20956 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.496505 21729 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.504802 20876 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.501469 21689 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.515753 21779 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.527905 20876 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:19.535197 21729 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.550547 21726 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.555452 20874 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.560021 21764 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.559911 20777 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:19.562405 20950 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.576656 21688 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:19.576996 20952 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:19.579813 21776 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:19.583952 21536 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.588304 21543 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.595355 21688 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.615599 20952 raft_consensus.cc:2932] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.622624 21729 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.629278 21690 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.636814 20767 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc05b4289a334b34a5f571e608bc9191 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:19.643796 21030 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.650511 21687 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:19.660737 21726 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.664629 20780 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:19.667903 20779 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:19.667017 21689 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.681273 20956 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.682380 20954 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:19.688232 21764 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:19.693748 21779 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.710558 20956 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.720788 20778 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.724699 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9178d1bf59624e7f8948c33552e74a00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:19.729892 21024 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.710942 21688 raft_consensus.cc:2932] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.736529 20880 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.752480 20950 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:19.756336 21018 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05b4289a334b34a5f571e608bc9191 with delete_type TABLET_DATA_TOMBSTONED (TS 7610624a16b44896b6aacea1928ab20e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47936
I20240506 15:14:19.795136 21764 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:19.804001 21729 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.809736 21690 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:19.812783 21726 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:19.813524 21796 tablet_replica.cc:330] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: stopping tablet replica
I20240506 15:14:19.806329 20778 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.817222 21690 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.831104 20950 raft_consensus.cc:2932] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.839396 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc90d9cec20d4d219e474797b7115172 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:19.843704 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet 9178d1bf59624e7f8948c33552e74a00 with delete_type TABLET_DATA_TOMBSTONED (TS 00fe71c1675441379f5f2c7490f9e63a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50460
I20240506 15:14:19.853474 21796 raft_consensus.cc:2229] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:19.854236 21796 raft_consensus.cc:2258] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:19.857435 21796 ts_tablet_manager.cc:1875] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:19.863243 21024 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:19.873324 21690 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:19.875391 21764 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:19.881682 20876 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:19.885234 21726 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.886335 21690 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.888895 21764 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:19.895239 20876 raft_consensus.cc:2932] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.857928 20780 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.908725 21796 ts_tablet_manager.cc:1888] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:19.910499 21796 log.cc:1196] T bc05b4289a334b34a5f571e608bc9191 P 7610624a16b44896b6aacea1928ab20e: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/wals/bc05b4289a334b34a5f571e608bc9191
I20240506 15:14:19.912635 21797 tablet_replica.cc:330] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: stopping tablet replica
I20240506 15:14:19.927284 20764 catalog_manager.cc:4830] TS 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): tablet bc05b4289a334b34a5f571e608bc9191 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:19.929301 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet cc90d9cec20d4d219e474797b7115172 with delete_type TABLET_DATA_TOMBSTONED (TS 00fe71c1675441379f5f2c7490f9e63a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50460
I20240506 15:14:19.930652 21797 raft_consensus.cc:2229] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:19.931433 21797 raft_consensus.cc:2258] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:19.932083 20779 catalog_manager.cc:5484] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 4 to 5, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:19.935704 21797 ts_tablet_manager.cc:1875] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:19.936758 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:19.943850 20941 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 with delete_type TABLET_DATA_TOMBSTONED (TS ecededcd854842d4867fda9fa63a4b66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35794
I20240506 15:14:19.969403 21798 tablet_replica.cc:330] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: stopping tablet replica
I20240506 15:14:19.970168 21798 raft_consensus.cc:2229] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:19.970814 21798 raft_consensus.cc:2258] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:19.975454 21798 ts_tablet_manager.cc:1875] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:19.988683 21797 ts_tablet_manager.cc:1888] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:19.989221 21797 log.cc:1196] T 9178d1bf59624e7f8948c33552e74a00 P 00fe71c1675441379f5f2c7490f9e63a: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/wals/9178d1bf59624e7f8948c33552e74a00
I20240506 15:14:19.991128 21797 tablet_replica.cc:330] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: stopping tablet replica
I20240506 15:14:19.991102 20767 catalog_manager.cc:4830] TS 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): tablet 9178d1bf59624e7f8948c33552e74a00 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:19.992537 21797 raft_consensus.cc:2229] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:19.993193 21797 pending_rounds.cc:62] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Trying to abort 1 pending ops.
I20240506 15:14:19.993446 21797 pending_rounds.cc:68] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Aborting op as it isn't in flight: id { term: 3 index: 4 } timestamp: 7024674649666772992 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cc90d9cec20d4d219e474797b7115172" old_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } } new_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } } }
I20240506 15:14:19.994814 21797 raft_consensus.cc:2894] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted
I20240506 15:14:19.995245 21797 raft_consensus.cc:2866] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240506 15:14:19.995715 21797 raft_consensus.cc:2258] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:19.996075 21798 ts_tablet_manager.cc:1888] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240506 15:14:19.996598 21798 log.cc:1196] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P ecededcd854842d4867fda9fa63a4b66: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/wals/5f7a6eab190f4de2bbd9e6b2ff60cdc1
I20240506 15:14:19.998648 20765 catalog_manager.cc:4830] TS ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): tablet 5f7a6eab190f4de2bbd9e6b2ff60cdc1 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:20.000741 21797 ts_tablet_manager.cc:1875] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:20.014060 21797 ts_tablet_manager.cc:1888] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240506 15:14:20.014406 21797 log.cc:1196] T cc90d9cec20d4d219e474797b7115172 P 00fe71c1675441379f5f2c7490f9e63a: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/wals/cc90d9cec20d4d219e474797b7115172
I20240506 15:14:20.015549 20767 catalog_manager.cc:4830] TS 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): tablet cc90d9cec20d4d219e474797b7115172 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:21.030530 21030 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.039528 20950 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:21.041090 20880 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240506 15:14:21.042441 20996 consensus_peers.cc:485] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faf3a7304753470787006e0ba7cf8852. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:21.042937 21726 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:21.048631 21726 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:21.072548 21726 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.081044 20880 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.087000 20876 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.092756 20950 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.095479 20779 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:21.119673 20846 consensus_peers.cc:485] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d84d513d60b64cba9c6bc3f8077e6b73. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:21.121944 21030 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:21.123301 20956 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:21.124646 21536 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:21.125970 21543 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:21.136533 21536 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.139765 21032 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.150503 20880 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.167809 21030 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:21.169257 21543 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:21.176818 20950 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.206861 20949 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240506 15:14:21.216933 20846 consensus_peers.cc:485] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e60758583b247c2946eb30ea2681286. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:21.218426 21813 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:21.228533 20778 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:21.282451 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 92687.7us
I20240506 15:14:21.293768 21813 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.299096 20950 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.321381 21031 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.326944 20778 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:21.329414 20996 consensus_peers.cc:485] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b648042628a4aec948d269392eb5cf5. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:21.329444 20880 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240506 15:14:21.336056 21030 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.331810 21764 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:21.341887 20949 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240506 15:14:21.345201 21764 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:21.353646 21726 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.362206 20950 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.380245 20778 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 2 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:21.457943 20874 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:21.493345 21824 ts_tablet_manager.cc:902] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:21.497462 21824 tablet_copy_client.cc:316] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:21.499336 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet faf3a7304753470787006e0ba7cf8852 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:21.499969 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet faf3a7304753470787006e0ba7cf8852 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-faf3a7304753470787006e0ba7cf8852
I20240506 15:14:21.519245 21042 tablet_copy_source_session.cc:205] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:21.534307 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf3a7304753470787006e0ba7cf8852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:21.565606 21824 tablet_copy_client.cc:791] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:21.567368 21824 tablet_copy_client.cc:655] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:21.574038 21824 tablet_copy_client.cc:531] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:21.583241 21824 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:21.611267 21824 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:21.612185 21824 tablet_bootstrap.cc:492] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:21.612856 21824 ts_tablet_manager.cc:1367] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20240506 15:14:21.615612 21824 raft_consensus.cc:348] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.616537 21824 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:21.617224 21824 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.626952 21824 ts_tablet_manager.cc:1398] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20240506 15:14:21.630560 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-faf3a7304753470787006e0ba7cf8852 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:21.631006 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-faf3a7304753470787006e0ba7cf8852 on tablet faf3a7304753470787006e0ba7cf8852 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:21.664321 21824 ts_tablet_manager.cc:902] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:21.684926 21824 tablet_copy_client.cc:316] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.129:44145
I20240506 15:14:21.686640 20893 tablet_copy_service.cc:140] P 00fe71c1675441379f5f2c7490f9e63a: Received BeginTabletCopySession request for tablet 7e60758583b247c2946eb30ea2681286 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49818)
I20240506 15:14:21.687238 20893 tablet_copy_service.cc:161] P 00fe71c1675441379f5f2c7490f9e63a: Beginning new tablet copy session on tablet 7e60758583b247c2946eb30ea2681286 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49818: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-7e60758583b247c2946eb30ea2681286
I20240506 15:14:21.694846 20893 tablet_copy_source_session.cc:205] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:21.699043 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e60758583b247c2946eb30ea2681286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:21.720613 21824 tablet_copy_client.cc:791] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:21.721319 21824 tablet_copy_client.cc:655] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:21.725152 21824 tablet_copy_client.cc:531] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:21.735277 21824 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:21.757886 21831 ts_tablet_manager.cc:902] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:21.763764 21824 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:21.764930 21824 tablet_bootstrap.cc:492] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:21.766055 21824 ts_tablet_manager.cc:1367] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.013s
I20240506 15:14:21.769181 21824 raft_consensus.cc:348] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.770124 21824 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:21.770951 21824 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.773902 21824 ts_tablet_manager.cc:1398] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240506 15:14:21.776165 21831 tablet_copy_client.cc:316] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.129:44145
I20240506 15:14:21.778272 20892 tablet_copy_service.cc:140] P 00fe71c1675441379f5f2c7490f9e63a: Received BeginTabletCopySession request for tablet d84d513d60b64cba9c6bc3f8077e6b73 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49818)
I20240506 15:14:21.778451 20893 tablet_copy_service.cc:342] P 00fe71c1675441379f5f2c7490f9e63a: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-7e60758583b247c2946eb30ea2681286 received from {username='slave'} at 127.0.0.1:49818
I20240506 15:14:21.778988 20893 tablet_copy_service.cc:433] P 00fe71c1675441379f5f2c7490f9e63a: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-7e60758583b247c2946eb30ea2681286 on tablet 7e60758583b247c2946eb30ea2681286 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:21.780102 20892 tablet_copy_service.cc:161] P 00fe71c1675441379f5f2c7490f9e63a: Beginning new tablet copy session on tablet d84d513d60b64cba9c6bc3f8077e6b73 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49818: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-d84d513d60b64cba9c6bc3f8077e6b73
I20240506 15:14:21.790570 20892 tablet_copy_source_session.cc:205] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:21.796084 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84d513d60b64cba9c6bc3f8077e6b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:21.821769 21831 tablet_copy_client.cc:791] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:21.822609 21831 tablet_copy_client.cc:655] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:21.826228 21831 tablet_copy_client.cc:531] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:21.836151 21831 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:21.852406 21824 ts_tablet_manager.cc:902] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:21.882910 21824 tablet_copy_client.cc:316] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:21.884503 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet 5b648042628a4aec948d269392eb5cf5 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:21.885064 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet 5b648042628a4aec948d269392eb5cf5 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-5b648042628a4aec948d269392eb5cf5
I20240506 15:14:21.894332 21831 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:21.896436 21831 tablet_bootstrap.cc:492] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:21.897742 21831 ts_tablet_manager.cc:1367] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.062s user 0.029s sys 0.016s
I20240506 15:14:21.904162 21831 raft_consensus.cc:348] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.905316 21831 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:21.905346 21042 tablet_copy_source_session.cc:205] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:21.906308 21831 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:21.908516 21831 ts_tablet_manager.cc:1398] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240506 15:14:21.908787 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b648042628a4aec948d269392eb5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:21.956403 20892 tablet_copy_service.cc:342] P 00fe71c1675441379f5f2c7490f9e63a: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-d84d513d60b64cba9c6bc3f8077e6b73 received from {username='slave'} at 127.0.0.1:49818
I20240506 15:14:21.957573 20892 tablet_copy_service.cc:433] P 00fe71c1675441379f5f2c7490f9e63a: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-d84d513d60b64cba9c6bc3f8077e6b73 on tablet d84d513d60b64cba9c6bc3f8077e6b73 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:22.027582 21824 tablet_copy_client.cc:791] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:22.028666 21824 tablet_copy_client.cc:655] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:22.044798 21824 tablet_copy_client.cc:531] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:22.057727 21824 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:22.085913 21690 raft_consensus.cc:1203] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:22.120523 21824 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:22.121513 21824 tablet_bootstrap.cc:492] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:22.122169 21824 ts_tablet_manager.cc:1367] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.065s user 0.029s sys 0.005s
I20240506 15:14:22.125299 21824 raft_consensus.cc:348] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:22.126837 21824 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:22.127579 21824 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:22.131714 21824 ts_tablet_manager.cc:1398] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240506 15:14:22.133908 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-5b648042628a4aec948d269392eb5cf5 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:22.134409 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-5b648042628a4aec948d269392eb5cf5 on tablet 5b648042628a4aec948d269392eb5cf5 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:22.247648 21690 raft_consensus.cc:1203] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:22.260811 21690 raft_consensus.cc:1203] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:22.436154 21690 raft_consensus.cc:1203] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240506 15:14:22.550269 21838 raft_consensus.cc:1050] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:22.561208 21838 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:22.580044 21690 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.580204 20950 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.585530 21822 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:22.587599 21822 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:22.620563 20880 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.622211 21838 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:22.632045 21841 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.638372 20950 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.647627 20880 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.661273 21690 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.667672 21837 raft_consensus.cc:1050] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:22.669955 21837 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:22.683775 20779 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.699687 21032 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.700547 21690 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.701951 21837 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:22.702577 20950 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:22.703063 21543 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:22.706080 21543 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:22.735024 21543 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.748301 21032 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.757632 20949 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.769810 21690 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.839767 21032 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:22.847467 20877 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:22.849671 21845 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:22.850044 21690 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:22.825320 20777 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.852464 21845 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:22.859897 21543 raft_consensus.cc:1050] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:22.861207 21838 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.862700 21543 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:22.867326 21690 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.877722 20880 raft_consensus.cc:2932] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.887099 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet faf3a7304753470787006e0ba7cf8852 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:22.902935 20874 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:22.903983 20764 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet faf3a7304753470787006e0ba7cf8852 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240506 15:14:22.927948 21689 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:22.930331 21031 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:22.932284 21813 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:22.935223 21837 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:22.950158 21536 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.961782 21031 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.989838 21822 raft_consensus.cc:1050] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:22.992812 21822 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:22.997920 20777 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:22.985631 21689 raft_consensus.cc:2932] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.023118 20767 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e60758583b247c2946eb30ea2681286 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:23.032331 20941 tablet_service.cc:1514] Processing DeleteTablet for tablet faf3a7304753470787006e0ba7cf8852 with delete_type TABLET_DATA_TOMBSTONED (TS ecededcd854842d4867fda9fa63a4b66 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35794
I20240506 15:14:23.040570 20949 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 15:14:23.042407 20880 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 15:14:23.043025 21690 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 15:14:23.044461 21845 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:23.045228 21032 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:23.047066 21543 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:23.047341 21838 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:23.048336 21845 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:23.051577 21690 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:23.058517 21856 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:23.064981 20950 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:23.069671 21857 tablet_replica.cc:330] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: stopping tablet replica
I20240506 15:14:23.092067 20777 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:23.097240 21813 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:23.124099 21857 raft_consensus.cc:2229] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:23.125046 21857 raft_consensus.cc:2258] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:23.129268 21857 ts_tablet_manager.cc:1875] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240506 15:14:23.105549 21855 log.cc:924] Time spent T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20240506 15:14:23.139557 21726 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.140691 21690 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.156666 20941 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e60758583b247c2946eb30ea2681286 with delete_type TABLET_DATA_TOMBSTONED (TS ecededcd854842d4867fda9fa63a4b66 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35794
I20240506 15:14:23.161211 20949 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
W20240506 15:14:23.174116 21859 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240506 15:14:23.176622 21857 ts_tablet_manager.cc:1888] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:23.177634 21857 log.cc:1196] T faf3a7304753470787006e0ba7cf8852 P ecededcd854842d4867fda9fa63a4b66: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/wals/faf3a7304753470787006e0ba7cf8852
I20240506 15:14:23.179459 20765 catalog_manager.cc:4830] TS ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): tablet faf3a7304753470787006e0ba7cf8852 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:23.179893 21857 tablet_replica.cc:330] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: stopping tablet replica
I20240506 15:14:23.181258 21857 raft_consensus.cc:2229] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:23.181980 21857 raft_consensus.cc:2258] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:23.167660 20780 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:23.200944 21857 ts_tablet_manager.cc:1875] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:23.205296 20877 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.257680 21857 ts_tablet_manager.cc:1888] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:23.258189 21857 log.cc:1196] T 7e60758583b247c2946eb30ea2681286 P ecededcd854842d4867fda9fa63a4b66: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/wals/7e60758583b247c2946eb30ea2681286
I20240506 15:14:23.260777 20765 catalog_manager.cc:4830] TS ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): tablet 7e60758583b247c2946eb30ea2681286 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:23.265628 21031 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:23.281445 21690 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:23.282106 20950 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:23.287115 21536 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.288659 21863 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:23.297268 21726 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:23.300149 21863 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.303297 20949 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.299726 21690 raft_consensus.cc:2932] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.314647 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b648042628a4aec948d269392eb5cf5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:23.320770 21689 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.342115 20777 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.347106 20950 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.361689 21030 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.362653 20780 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New cstate: current_term: 2 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.398382 20880 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:23.408124 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b648042628a4aec948d269392eb5cf5 with delete_type TABLET_DATA_TOMBSTONED (TS 00fe71c1675441379f5f2c7490f9e63a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50460
I20240506 15:14:23.419021 21690 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:23.433797 21868 tablet_replica.cc:330] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: stopping tablet replica
I20240506 15:14:23.435127 21868 raft_consensus.cc:2229] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:23.436070 21868 raft_consensus.cc:2258] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:23.449723 21813 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:23.455566 20950 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:23.462051 21813 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:23.470011 21856 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.474810 21868 ts_tablet_manager.cc:1875] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:23.482165 21690 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.487856 20949 raft_consensus.cc:2932] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.500694 20767 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d84d513d60b64cba9c6bc3f8077e6b73 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:23.501125 21868 ts_tablet_manager.cc:1888] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240506 15:14:23.504969 20779 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:23.512326 21868 log.cc:1196] T 5b648042628a4aec948d269392eb5cf5 P 00fe71c1675441379f5f2c7490f9e63a: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/wals/5b648042628a4aec948d269392eb5cf5
I20240506 15:14:23.514938 20767 catalog_manager.cc:4830] TS 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): tablet 5b648042628a4aec948d269392eb5cf5 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:23.523249 21018 tablet_service.cc:1514] Processing DeleteTablet for tablet d84d513d60b64cba9c6bc3f8077e6b73 with delete_type TABLET_DATA_TOMBSTONED (TS 7610624a16b44896b6aacea1928ab20e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47936
I20240506 15:14:23.542863 21870 tablet_replica.cc:330] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: stopping tablet replica
I20240506 15:14:23.544226 21870 raft_consensus.cc:2229] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:23.545250 21870 raft_consensus.cc:2258] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:23.550136 21870 ts_tablet_manager.cc:1875] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:23.569209 21870 ts_tablet_manager.cc:1888] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:23.569725 21870 log.cc:1196] T d84d513d60b64cba9c6bc3f8077e6b73 P 7610624a16b44896b6aacea1928ab20e: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/wals/d84d513d60b64cba9c6bc3f8077e6b73
I20240506 15:14:23.571692 20764 catalog_manager.cc:4830] TS 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): tablet d84d513d60b64cba9c6bc3f8077e6b73 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:24.638689 20949 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:24.646436 21030 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:24.648769 20880 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:24.651103 21444 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:24.652428 21869 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:24.679899 21444 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
W20240506 15:14:24.686439 20922 consensus_peers.cc:485] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93fe2d6a721f4eaeb4b3f4b5f71e1a91. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:24.689507 21031 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.721647 20880 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.734998 20877 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:24.735953 20778 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:24.747933 20846 consensus_peers.cc:485] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48f44ddbf0044d1c9c94508ba5aa2406. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:24.749285 20950 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:24.749099 21031 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:24.750721 21536 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:24.752125 21536 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:24.760577 21536 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.763720 20950 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.765308 21031 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.789749 21024 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:24.787895 20778 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.820056 20877 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:24.821540 21863 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:24.833181 20950 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 15:14:24.834923 21866 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20240506 15:14:24.842953 20996 consensus_peers.cc:485] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b96eaaaebde34390bee0dafdfb156080. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:24.852273 21890 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.854692 20950 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.866755 20877 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.872308 20949 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:24.874795 20779 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index -1 to 2, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:24.883911 20922 consensus_peers.cc:485] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81702cae5854497d947832321f954762. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:24.884215 21031 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:24.885990 21869 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:24.886951 20877 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:24.889218 21869 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:24.917738 21869 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.919195 21031 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.922029 20877 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:14:24.940387 20777 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:25.227106 21896 ts_tablet_manager.cc:902] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:25.228818 21896 tablet_copy_client.cc:316] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.130:40545
I20240506 15:14:25.235450 21900 ts_tablet_manager.cc:902] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:14:25.239598 21900 tablet_copy_client.cc:316] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.129:44145
I20240506 15:14:25.241268 20892 tablet_copy_service.cc:140] P 00fe71c1675441379f5f2c7490f9e63a: Received BeginTabletCopySession request for tablet 48f44ddbf0044d1c9c94508ba5aa2406 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49818)
I20240506 15:14:25.241834 20892 tablet_copy_service.cc:161] P 00fe71c1675441379f5f2c7490f9e63a: Beginning new tablet copy session on tablet 48f44ddbf0044d1c9c94508ba5aa2406 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49818: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-48f44ddbf0044d1c9c94508ba5aa2406
I20240506 15:14:25.249491 20892 tablet_copy_source_session.cc:205] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:25.252727 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f44ddbf0044d1c9c94508ba5aa2406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:25.254045 20968 tablet_copy_service.cc:140] P ecededcd854842d4867fda9fa63a4b66: Received BeginTabletCopySession request for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:44378)
I20240506 15:14:25.254685 20968 tablet_copy_service.cc:161] P ecededcd854842d4867fda9fa63a4b66: Beginning new tablet copy session on tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:44378: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-93fe2d6a721f4eaeb4b3f4b5f71e1a91
I20240506 15:14:25.262226 20968 tablet_copy_source_session.cc:205] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:25.265096 21896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:25.270838 21900 tablet_copy_client.cc:791] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:25.271452 21900 tablet_copy_client.cc:655] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:25.276877 21900 tablet_copy_client.cc:531] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:25.286888 21900 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:25.301590 21896 tablet_copy_client.cc:791] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:25.303925 21896 tablet_copy_client.cc:655] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:25.308584 21902 ts_tablet_manager.cc:902] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:25.309754 21896 tablet_copy_client.cc:531] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:25.320618 21905 ts_tablet_manager.cc:902] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:25.325160 21900 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:25.326099 21900 tablet_bootstrap.cc:492] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:25.326902 21900 ts_tablet_manager.cc:1367] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.005s
I20240506 15:14:25.329731 21900 raft_consensus.cc:348] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.330686 21900 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:25.331324 21900 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.333739 21902 tablet_copy_client.cc:316] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:14:25.337790 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet b96eaaaebde34390bee0dafdfb156080 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:14:25.338630 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet b96eaaaebde34390bee0dafdfb156080 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-b96eaaaebde34390bee0dafdfb156080
I20240506 15:14:25.363703 21042 tablet_copy_source_session.cc:205] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:25.371577 21902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96eaaaebde34390bee0dafdfb156080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:25.331315 21896 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:25.393226 21905 tablet_copy_client.cc:316] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.130:40545
I20240506 15:14:25.396852 20968 tablet_copy_service.cc:140] P ecededcd854842d4867fda9fa63a4b66: Received BeginTabletCopySession request for tablet 81702cae5854497d947832321f954762 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:44378)
I20240506 15:14:25.397775 20968 tablet_copy_service.cc:161] P ecededcd854842d4867fda9fa63a4b66: Beginning new tablet copy session on tablet 81702cae5854497d947832321f954762 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:44378: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-81702cae5854497d947832321f954762
I20240506 15:14:25.404467 21902 tablet_copy_client.cc:791] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:25.405144 21902 tablet_copy_client.cc:655] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:25.409538 21902 tablet_copy_client.cc:531] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:25.423290 21902 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:25.431241 21896 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:25.432304 21896 tablet_bootstrap.cc:492] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:25.433029 21896 ts_tablet_manager.cc:1367] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.102s user 0.028s sys 0.004s
I20240506 15:14:25.436643 21896 raft_consensus.cc:348] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.437702 21896 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:25.438519 21896 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.454322 21896 ts_tablet_manager.cc:1398] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.021s user 0.009s sys 0.004s
I20240506 15:14:25.454825 20968 tablet_copy_source_session.cc:205] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:25.456293 20967 tablet_copy_service.cc:342] P ecededcd854842d4867fda9fa63a4b66: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-93fe2d6a721f4eaeb4b3f4b5f71e1a91 received from {username='slave'} at 127.0.0.1:44378
I20240506 15:14:25.456669 20967 tablet_copy_service.cc:433] P ecededcd854842d4867fda9fa63a4b66: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-93fe2d6a721f4eaeb4b3f4b5f71e1a91 on tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:25.459522 21902 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:25.460403 21902 tablet_bootstrap.cc:492] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:25.461077 21902 ts_tablet_manager.cc:1367] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.038s user 0.030s sys 0.001s
I20240506 15:14:25.464107 21902 raft_consensus.cc:348] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.464991 21902 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:25.465602 21902 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.468493 21902 ts_tablet_manager.cc:1398] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240506 15:14:25.469741 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-b96eaaaebde34390bee0dafdfb156080 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:14:25.470070 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-b96eaaaebde34390bee0dafdfb156080 on tablet b96eaaaebde34390bee0dafdfb156080 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:25.474651 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81702cae5854497d947832321f954762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:25.489177 21900 ts_tablet_manager.cc:1398] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.162s user 0.056s sys 0.069s
I20240506 15:14:25.490866 20892 tablet_copy_service.cc:342] P 00fe71c1675441379f5f2c7490f9e63a: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-48f44ddbf0044d1c9c94508ba5aa2406 received from {username='slave'} at 127.0.0.1:49818
I20240506 15:14:25.491263 20892 tablet_copy_service.cc:433] P 00fe71c1675441379f5f2c7490f9e63a: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-48f44ddbf0044d1c9c94508ba5aa2406 on tablet 48f44ddbf0044d1c9c94508ba5aa2406 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:25.508199 21905 tablet_copy_client.cc:791] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:25.512221 21905 tablet_copy_client.cc:655] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:25.518105 21905 tablet_copy_client.cc:531] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:25.530645 21905 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:25.550093 21905 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:14:25.550786 21905 tablet_bootstrap.cc:492] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:25.551690 21905 ts_tablet_manager.cc:1367] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20240506 15:14:25.554152 21905 raft_consensus.cc:348] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.554780 21905 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:25.555512 21905 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:25.557793 21905 ts_tablet_manager.cc:1398] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240506 15:14:25.559597 20968 tablet_copy_service.cc:342] P ecededcd854842d4867fda9fa63a4b66: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-81702cae5854497d947832321f954762 received from {username='slave'} at 127.0.0.1:44378
I20240506 15:14:25.560005 20968 tablet_copy_service.cc:433] P ecededcd854842d4867fda9fa63a4b66: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-81702cae5854497d947832321f954762 on tablet 81702cae5854497d947832321f954762 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:25.674501 21689 raft_consensus.cc:1203] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:25.737758 21689 raft_consensus.cc:1203] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240506 15:14:25.749905 21689 raft_consensus.cc:1203] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:14:25.785436 21689 raft_consensus.cc:1203] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:14:26.133093 21912 raft_consensus.cc:1050] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:26.135471 21912 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.141898 21689 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.142922 21024 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.143496 21536 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:26.144604 21912 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:26.144826 20956 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.152129 21536 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:26.157124 21888 raft_consensus.cc:1050] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:26.159981 21888 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.161711 21536 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.163744 20956 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.172446 21689 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.177388 21690 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.179278 20949 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.179548 21866 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:26.181656 21863 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:26.184285 20880 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 15:14:26.186285 21863 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:26.216729 21879 raft_consensus.cc:1050] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:26.198664 20777 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:26.219252 21879 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.223912 21921 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.257251 20880 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.259517 21031 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.271227 21689 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.280812 20949 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.290028 21913 raft_consensus.cc:1050] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:14:26.298305 21913 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.328873 20780 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index 2 to 3, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:26.335165 21689 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.336115 21031 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.337308 20877 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.338850 21913 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.339756 21444 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.341934 21869 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.349267 20874 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.350780 21879 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.351543 21031 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.353127 21869 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.355345 21689 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:14:26.357964 21913 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.365298 21444 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.377068 21690 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.381017 20874 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.382079 21879 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.385473 21031 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.409729 21032 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.416944 20877 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.417464 21690 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.431620 20776 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.447775 21031 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.448194 20874 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.454896 21690 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:26.456540 20949 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:26.455829 20779 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:26.457073 21863 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.458482 21888 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.472203 21920 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.474197 21690 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.483961 20949 raft_consensus.cc:2932] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.484766 20950 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:26.489635 21689 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:26.500728 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b96eaaaebde34390bee0dafdfb156080 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:26.500667 21937 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:26.502295 21031 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:26.504496 20881 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:26.505548 21916 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:26.505441 20776 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New cstate: current_term: 1 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.510481 21689 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240506 15:14:26.512506 21536 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:26.513312 21879 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.516889 21536 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.523952 21937 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:26.529317 21689 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.532883 20765 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81702cae5854497d947832321f954762 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:26.533109 21688 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.535953 20881 raft_consensus.cc:2932] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.526206 21032 raft_consensus.cc:2932] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.553622 20767 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48f44ddbf0044d1c9c94508ba5aa2406 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:26.560158 20778 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.567346 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet b96eaaaebde34390bee0dafdfb156080 with delete_type TABLET_DATA_TOMBSTONED (TS 00fe71c1675441379f5f2c7490f9e63a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50460
I20240506 15:14:26.592234 21018 tablet_service.cc:1514] Processing DeleteTablet for tablet 81702cae5854497d947832321f954762 with delete_type TABLET_DATA_TOMBSTONED (TS 7610624a16b44896b6aacea1928ab20e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47936
I20240506 15:14:26.597079 21942 tablet_replica.cc:330] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: stopping tablet replica
I20240506 15:14:26.597955 21941 tablet_replica.cc:330] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: stopping tablet replica
I20240506 15:14:26.605295 20776 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index 3 to 4, VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) evicted. New cstate: current_term: 1 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:26.619232 21942 raft_consensus.cc:2229] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:26.620157 21942 raft_consensus.cc:2258] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:26.623373 21942 ts_tablet_manager.cc:1875] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:26.626168 21941 raft_consensus.cc:2229] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:26.627102 21941 raft_consensus.cc:2258] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:26.631613 21941 ts_tablet_manager.cc:1875] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:26.639493 21942 ts_tablet_manager.cc:1888] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240506 15:14:26.640071 21942 log.cc:1196] T 81702cae5854497d947832321f954762 P 7610624a16b44896b6aacea1928ab20e: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/wals/81702cae5854497d947832321f954762
I20240506 15:14:26.641716 20764 catalog_manager.cc:4830] TS 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): tablet 81702cae5854497d947832321f954762 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:26.642736 20941 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f44ddbf0044d1c9c94508ba5aa2406 with delete_type TABLET_DATA_TOMBSTONED (TS ecededcd854842d4867fda9fa63a4b66 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35794
I20240506 15:14:26.656761 21943 tablet_replica.cc:330] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: stopping tablet replica
I20240506 15:14:26.657691 21943 raft_consensus.cc:2229] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:26.658324 21943 raft_consensus.cc:2258] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:26.661995 21943 ts_tablet_manager.cc:1875] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:26.662184 21941 ts_tablet_manager.cc:1888] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:26.662657 21941 log.cc:1196] T b96eaaaebde34390bee0dafdfb156080 P 00fe71c1675441379f5f2c7490f9e63a: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/wals/b96eaaaebde34390bee0dafdfb156080
I20240506 15:14:26.664340 20767 catalog_manager.cc:4830] TS 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): tablet b96eaaaebde34390bee0dafdfb156080 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:26.677207 21943 ts_tablet_manager.cc:1888] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240506 15:14:26.677786 21943 log.cc:1196] T 48f44ddbf0044d1c9c94508ba5aa2406 P ecededcd854842d4867fda9fa63a4b66: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-1-root/wals/48f44ddbf0044d1c9c94508ba5aa2406
I20240506 15:14:26.679315 20765 catalog_manager.cc:4830] TS ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): tablet 48f44ddbf0044d1c9c94508ba5aa2406 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:14:28.494551 21949 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.495188 21949 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.502547 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 682) took 1658 ms. Trace:
I20240506 15:14:28.502965 20874 rpcz_store.cc:276] 0506 15:14:26.844656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:28.501738 (+1657082us) service_pool.cc:225] Handling call
0506 15:14:28.502089 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:28.502247 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:28.502276 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:28.502310 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:28.502339 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:28.502391 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:28.502492 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:28.503808 20845 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.007s sys 0.004s
W20240506 15:14:28.504137 20845 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.580s user 0.007s sys 0.004s
I20240506 15:14:28.504290 21952 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.505002 21952 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.497412 21950 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.507139 21950 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.507931 21951 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.508513 21951 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.509315 21950 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.510552 21951 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.517470 21950 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.518121 21950 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.522228 21950 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.521948 21952 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.526891 21950 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.523483 21951 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.520354 21949 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.511201 21948 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.526396 21952 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.529532 21953 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.534511 21951 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.539700 21951 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.533603 21950 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.542183 21950 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.543121 21949 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.543862 21949 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.531622 21946 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.553329 21946 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.556720 21946 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.560058 21949 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.546043 21953 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.560370 21951 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.561471 21950 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.561630 21958 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.534551 21948 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.562623 21955 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.564855 21959 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.565246 21949 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.568449 21949 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.560295 21957 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.569450 21957 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.570567 21949 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.570919 21948 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.571364 21957 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.567505 21955 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.566593 21950 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.573758 21955 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.565246 21951 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.565356 21960 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.574441 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 685) took 1576 ms. Trace:
I20240506 15:14:28.536953 21952 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.575047 21960 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.575578 21951 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.575044 20876 rpcz_store.cc:276] 0506 15:14:26.998327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:28.573257 (+1574930us) service_pool.cc:225] Handling call
0506 15:14:28.573709 (+ 452us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:28.573898 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:28.573934 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:28.573986 (+ 52us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:28.574028 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:28.574093 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:28.574232 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:28.574474 21950 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:28.585048 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.015s sys 0.005s
W20240506 15:14:28.585418 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.268s user 0.015s sys 0.005s
I20240506 15:14:28.586355 21953 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.586572 21960 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.586828 21952 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.576920 21959 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.566874 21946 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.577898 21958 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.590142 21954 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.589072 21953 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.598692 21949 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.599282 21949 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.603916 21959 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.590921 21946 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.605527 21949 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.604110 21953 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.606299 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 683) took 1750 ms. Trace:
I20240506 15:14:28.606437 21946 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.592887 21954 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.608399 20881 rpcz_store.cc:276] 0506 15:14:26.855906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:28.605234 (+1749328us) service_pool.cc:225] Handling call
0506 15:14:28.605660 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:28.605870 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:28.605909 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:28.605967 (+ 58us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:28.606011 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:28.606081 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:28.606228 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:28.614579 21954 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.616322 21958 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.618561 21946 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.619145 21946 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.619660 21953 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.621618 21946 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.623136 21965 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.623732 21965 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.625873 21965 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.610872 21949 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.633807 21954 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.634268 21959 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.634361 21954 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.634812 21959 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.632685 21949 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.638662 21964 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.639166 21949 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.636906 21959 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.639269 21952 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.640862 21952 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.642715 21952 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.643187 21952 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.643612 21952 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.646510 21952 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.646852 21950 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.643615 21960 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.647424 21950 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.627638 21963 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.648470 21963 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.649075 21955 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.650218 21950 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.649829 21955 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.650512 21963 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.650774 21950 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.651031 21963 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.651271 21950 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.651494 21963 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.621552 21958 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.652433 21957 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.652731 21958 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.653245 21950 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.653185 21957 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.654596 21954 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.655385 21947 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.657105 21962 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.657658 21962 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.658093 21955 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.655929 21957 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.655898 21947 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.658859 21966 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.660717 21947 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.661042 21946 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.639273 21964 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.661657 21946 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.663734 21964 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.663810 21958 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.664089 21946 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.661108 21966 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.664389 21957 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.665333 21957 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.660566 21962 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.666705 21963 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.669942 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.670192 21961 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.670742 21689 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:28.670809 21961 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.667618 21947 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.667176 21957 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.672449 21947 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.673080 21961 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.648010 21960 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.667817 21962 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.676281 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.677078 21689 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:28.674952 21947 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.678545 21962 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.681109 21954 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.681670 21954 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:28.683431 20998 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.366s user 0.008s sys 0.009s
I20240506 15:14:28.683578 21954 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.667229 21966 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.685798 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.686560 21689 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:28.653908 21950 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.687860 20922 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.688092 21950 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.689100 21966 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.689513 21966 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.689934 21966 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:28.690308 21950 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.681351 21962 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.690928 21957 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:28.683763 20998 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.366s user 0.008s sys 0.009s
I20240506 15:14:28.691208 20922 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.691468 21957 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.701431 21957 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:28.691829 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 1.3743e+06us
I20240506 15:14:28.706621 21944 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.707314 21944 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.709615 21956 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.710151 21956 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.712782 21921 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.713126 21947 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.713411 21921 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.713678 21947 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.714772 21944 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.715337 21921 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.715639 21947 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.718562 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.684118 21954 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.694232 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.721429 20881 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:28.721297 21954 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.722842 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.694182 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.723591 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.694082 21952 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.725052 21952 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.726306 21971 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:28.726760 21971 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.726926 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.727128 21971 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:28.727175 21952 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.695348 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.727656 21686 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:28.696278 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 684) took 1727 ms. Trace:
I20240506 15:14:28.696300 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.729303 21954 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.729489 20880 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:28.697829 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.729296 20877 rpcz_store.cc:276] 0506 15:14:26.969290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:28.695583 (+1726293us) service_pool.cc:225] Handling call
0506 15:14:28.695840 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:28.695984 (+ 144us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:28.696009 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:28.696050 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:28.696077 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:28.696124 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:28.696212 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:28.712217 21956 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.715349 21944 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.733065 21944 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.735253 21944 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.735117 21966 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.737354 21966 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.743932 21956 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.711944 21888 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.744503 21956 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.744845 21888 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.745090 21962 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.745852 21962 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.746479 21956 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.746913 21888 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.748075 21962 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.677361 21960 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.746518 21965 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.752652 21965 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.754590 21965 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.754917 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:28.755578 21689 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:28.757009 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.757738 21686 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:28.762594 21971 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.763397 21656 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:28.763787 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.764710 21954 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:28.764947 21971 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.765151 21954 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.765594 21954 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:28.772522 20979 debug-util.cc:398] Leaking SignalData structure 0x7b080016afa0 after lost signal to thread 19729
W20240506 15:14:28.773936 20979 debug-util.cc:398] Leaking SignalData structure 0x7b08000eede0 after lost signal to thread 20814
W20240506 15:14:28.774415 20979 debug-util.cc:398] Leaking SignalData structure 0x7b08001be5e0 after lost signal to thread 20828
W20240506 15:14:28.775588 20979 debug-util.cc:398] Leaking SignalData structure 0x7b0800316fc0 after lost signal to thread 20907
I20240506 15:14:28.719502 21947 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.776484 21947 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:14:28.776690 20979 debug-util.cc:398] Leaking SignalData structure 0x7b0800329ae0 after lost signal to thread 20982
I20240506 15:14:28.777072 21947 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:28.777976 20979 debug-util.cc:398] Leaking SignalData structure 0x7b080012a560 after lost signal to thread 21056
I20240506 15:14:28.777406 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:28.778636 21686 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:28.779115 20979 debug-util.cc:398] Leaking SignalData structure 0x7b0800300420 after lost signal to thread 21718
I20240506 15:14:28.699071 21970 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.780822 21970 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.783049 21970 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.785290 20922 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.788578 21961 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.786545 21971 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.788383 21967 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.790547 21961 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.791072 21971 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.791504 21967 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.792778 21960 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.792997 21971 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.792937 21961 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.789945 21956 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.785818 21965 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.793844 21967 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.794337 21956 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.794468 21965 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.787813 21966 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.796104 21888 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.793336 21960 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.796438 21965 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.796554 21966 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.798404 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.796689 21956 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.799116 21966 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.787515 21953 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.798444 21960 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.800591 21974 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.799134 21965 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:28.800415 21944 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.801120 21971 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.801501 21965 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.801633 21944 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.801893 21971 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.802809 21975 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.803682 21944 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.803392 21975 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.804093 21971 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.801497 21945 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.786250 21969 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.805121 21920 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.805236 21945 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.805454 21969 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.805653 21920 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.796664 21888 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.807554 21920 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.807687 21921 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.804193 21944 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.808202 21966 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.808223 21921 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.809422 21945 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.810062 21888 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.810609 21921 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.809422 21966 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.812228 21952 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.812419 21920 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.812872 21945 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.812867 21920 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.812773 21952 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.815495 21945 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.816990 21920 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.817279 21945 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.817734 21921 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.817941 21945 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.818075 21952 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.812770 21966 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.818485 21945 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.818248 21921 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.819329 21952 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.819489 21888 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.819882 21952 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.820013 21888 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.820616 21945 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.821254 21921 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.821770 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.822038 21888 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.822525 21952 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.841548 21888 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.822858 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.799090 21976 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.823218 21956 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.842258 21888 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.842742 21956 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.844645 21956 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.846091 21888 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.823695 21966 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.848630 20957 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:28.848706 21966 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.850705 21966 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.851306 21686 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:28.852681 21687 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:28.853900 21976 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.855226 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.856041 21966 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:28.856576 21966 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:28.857415 21966 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.860517 20949 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:28.801151 21974 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.824131 21920 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.861588 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.861716 21920 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.862849 21974 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.789831 21689 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:28.809144 21944 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.863687 21920 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.825398 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.826481 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.864795 21944 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.825459 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.865248 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.865909 21027 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.866009 21688 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:28.865674 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.866689 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.867085 21027 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.867796 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.825999 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.868294 20922 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:28.869025 21956 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:28.868976 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.827667 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.869529 21921 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.829571 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.869684 21030 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.828557 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.870678 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.870805 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.871006 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.871279 21030 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:28.871343 21023 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.871552 21028 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.830484 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.832492 21960 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.800451 21953 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.872402 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.872534 21026 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.872678 21024 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 1.
I20240506 15:14:28.872854 21960 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.873668 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.873973 20878 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 1.
I20240506 15:14:28.874310 21024 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.874146 21921 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.877801 21656 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:28.834231 21972 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.801976 21965 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:28.872269 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.879405 21028 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.880198 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.880702 21028 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.881347 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.881759 21028 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:28.856017 21976 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.882395 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.827560 21975 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.882802 21028 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.884500 21952 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.865422 21031 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 1.
I20240506 15:14:28.885344 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.885497 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.841774 21945 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.886421 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.886620 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.886332 21945 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.887522 21032 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:28.828617 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.869575 21956 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:28.875008 21960 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.875154 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.832201 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.887477 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.876283 21027 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:28.891999 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.892288 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.892616 21024 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.892727 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.892921 21027 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.893452 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.873600 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.876894 21953 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.873574 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.898188 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.876120 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:28.883165 21950 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.790099 21954 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.888985 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.890013 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.838317 21947 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.887671 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.891036 21025 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.890818 21956 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.818719 21971 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.869733 20954 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.807533 21969 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.872329 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.894335 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.895151 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.931354 21024 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.899328 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.898243 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.901358 21032 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:28.901440 21029 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.926071 21971 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.927336 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.914208 21965 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:28.928071 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.939543 21653 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:28.941005 21653 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:28.941915 20957 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:28.932266 21030 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:28.911621 21954 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.933027 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.899992 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:28.931259 21027 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
W20240506 15:14:28.928573 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 50147.6us
W20240506 15:14:28.944482 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44603.4us
I20240506 15:14:28.909777 21026 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.935456 21975 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.934710 21023 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:28.932940 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.947312 20956 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:28.934594 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.943397 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.945840 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.949049 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:28.909570 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.950170 21968 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:28.942580 20953 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:28.913033 21921 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.950084 20957 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:28.956676 21979 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.956969 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.957744 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.948472 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:28.930220 21028 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.895779 21952 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:28.954358 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.967671 20877 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:28.963443 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:28.964628 21980 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:28.965441 21971 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:28.969084 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.968166 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:28.966013 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:28.970820 20845 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:28.965977 21957 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.969888 20958 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 1.
I20240506 15:14:28.952371 21960 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.973225 21963 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.976382 21963 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.977030 21963 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:28.946679 21975 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.976306 21945 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.960717 21954 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:28.957216 21979 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.972333 21957 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.985725 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.981314 21975 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:28.981544 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.973729 21965 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:28.981333 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:28.981257 21946 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:28.981988 21025 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:28.988986 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.970165 21952 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:28.935009 21950 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:28.986820 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.989053 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.991376 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.981765 21974 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:28.999498 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.996548 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.989454 21957 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.027115 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.979725 21960 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.031109 21947 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:29.031476 21960 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.027782 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.031635 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.032893 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.034221 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.035346 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.036160 20957 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:28.998972 21958 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.032927 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.027112 21945 raft_consensus.cc:3032] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:29.034250 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.038144 20956 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:29.038348 21945 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:29.038996 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.911471 21972 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.039178 21945 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.030730 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:28.983489 21954 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.044438 21954 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.028353 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.040010 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.015527 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.040812 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.046406 20878 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:29.046782 20877 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:29.040491 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.048738 21964 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.049147 21964 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.003932 21979 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.049624 21964 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.030761 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.025569 21946 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:28.994642 21653 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:29.043375 21945 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.050153 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.042274 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.057261 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.057454 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.058178 21955 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.041086 21958 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.059846 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:29.060004 20845 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.059548 21955 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.060868 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.061664 20845 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.062294 21948 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.062369 21984 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.064530 21948 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.064731 21984 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.063043 20845 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.065053 21948 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.065224 21984 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.054114 21946 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.068398 20845 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.069587 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.071425 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.044157 21960 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.067144 21949 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.073105 21949 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.073558 21949 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.067596 21959 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:29.060756 21939 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.061172 21958 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.065996 21951 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.063269 21955 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.080229 20996 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:29.083210 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:28.989310 21980 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.088572 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.077906 21982 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.105s
I20240506 15:14:29.090027 21982 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:29.090381 21982 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:29.090746 21950 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.078053 21921 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240506 15:14:29.074438 21959 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.096446 20952 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.097151 21959 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.088091 21974 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.096487 21921 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:29.083925 21939 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.081838 21951 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:29.072294 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 87548.8us
I20240506 15:14:29.109908 21939 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.109917 21951 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.105326 21921 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:29.104243 21974 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.112059 21921 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.112641 21974 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.133236 21653 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:29.145608 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.105060 21950 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:29.091181 21982 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.157907 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.145563 21921 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.167618 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:29.155925 21968 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:29.172922 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:29.131556 21957 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.173094 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:29.175421 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:29.175714 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:29.163965 21975 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20240506 15:14:29.183269 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.183549 20996 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:29.173703 21968 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.158862 21950 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.199791 20878 raft_consensus.cc:1226] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.203711 20996 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:29.208318 21920 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.208875 21920 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.205034 21979 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:29.209311 21920 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.209688 21979 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.210239 21979 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.208531 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.193672 21950 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.192925 21968 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.234185 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.235515 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.205646 21960 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.225216 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.262753 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.274315 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.338757 20996 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:29.340060 21972 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.343030 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.344285 20952 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
W20240506 15:14:29.348500 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 280064us
I20240506 15:14:29.349458 20845 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.367973 20845 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:29.387068 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 48524.1us
I20240506 15:14:29.370456 21960 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.390772 20996 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:29.396569 20845 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.390403 21960 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.390707 21976 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:29.390174 21982 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.408831 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.414984 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.415300 21687 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240506 15:14:29.399050 21960 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.417258 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.419168 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.419865 20877 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:29.420773 21686 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:29.422539 21952 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.414008 21972 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.406373 21986 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.417635 21957 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.446635 21957 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.448123 21957 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.448544 21957 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.448876 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.439846 21970 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20240506 15:14:29.438314 21986 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.446846 21688 raft_consensus.cc:2379] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 2: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:29.456597 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.455587 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.456104 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.438409 21972 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:29.434190 21956 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240506 15:14:29.460045 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.461913 21972 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.428647 21964 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.472844 21954 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.473470 21964 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.459169 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.469336 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.462513 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.458547 21952 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.475185 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.499845 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.452708 21971 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:29.502166 21971 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:29.500139 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.500917 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.438759 21946 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.501618 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.502563 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.503275 20876 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:29.509905 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.454830 21980 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.452270 21986 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.519331 21954 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.521791 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.523733 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.525821 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.508327 21688 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.532302 21946 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.535166 21957 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.538924 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.544064 21968 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.546700 21690 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:29.451891 21963 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.548382 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:29.551525 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 155000us
I20240506 15:14:29.555105 20845 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.555999 20845 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:29.558276 21950 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.563086 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.568385 21960 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.568919 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.570477 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.571283 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.571638 21960 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.572244 21964 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.572602 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.578986 21968 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.579190 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.577854 21956 consensus_queue.cc:1046] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240506 15:14:29.567580 20776 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 2 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:29.582737 21956 raft_consensus.cc:3032] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:29.583088 21956 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:29.550102 21951 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.585918 21951 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.572659 21964 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.587997 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.591078 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.551026 21946 consensus_queue.cc:1046] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20240506 15:14:29.583626 20776 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:29.594170 21939 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.568078 21949 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.551688 21948 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.581437 21960 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240506 15:14:29.603314 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47411.6us
I20240506 15:14:29.560755 21984 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.604199 20845 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.593046 21964 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.606457 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.603528 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.609452 21949 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.600939 21972 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.611696 21972 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:29.611932 21958 consensus_meta.cc:287] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.007s
I20240506 15:14:29.583863 21956 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.613175 21958 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.614001 21956 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.615337 21958 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.616832 21946 raft_consensus.cc:3032] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:29.617242 21946 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:29.617436 21948 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.618191 21948 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.617967 21946 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.619174 21948 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.620465 21939 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:29.620594 21949 consensus_queue.cc:1159] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7610624a16b44896b6aacea1928ab20e" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20240506 15:14:29.603098 21974 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.624444 21974 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.624577 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.587256 21979 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.604414 21950 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.626513 21979 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.627735 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.630333 21686 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.635478 21957 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:29.637081 21955 consensus_meta.cc:287] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.549s user 0.006s sys 0.028s
I20240506 15:14:29.639725 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.641052 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.650599 21023 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:29.641372 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.641687 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.642879 21025 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:29.653260 21955 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:29.654565 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 79660.6us
I20240506 15:14:29.657069 21955 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.551996 21971 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.659013 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.603238 21960 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.661470 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.649026 21948 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.658340 21963 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.659857 21946 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.672062 21986 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.662251 21960 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.674669 21964 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:29.676086 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 58790.4us
I20240506 15:14:29.678359 21920 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.675503 21984 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.681093 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:29.682277 21959 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.584s user 0.025s sys 0.031s
I20240506 15:14:29.682674 21959 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.683877 21964 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.684420 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.688051 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.661919 21957 consensus_queue.cc:1046] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240506 15:14:29.689043 21951 raft_consensus.cc:3032] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:29.689400 21951 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:29.687625 20845 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.690979 21031 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:29.691802 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.690131 21951 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.692842 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.692914 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.695047 21939 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.695120 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.695494 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.696003 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.695564 21939 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.676028 21971 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.697209 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.701074 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.701467 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.701534 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.701915 20949 raft_consensus.cc:2396] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240506 15:14:29.702649 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.702777 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.703967 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.699008 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.704444 21957 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:29.691061 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.704839 21957 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.704916 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.705320 21957 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:29.705780 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.707046 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.699045 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.707631 21920 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:29.699939 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.708317 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.698585 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.698683 21939 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.699313 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.712574 21960 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.714306 21951 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.715173 21960 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:29.715961 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.696702 21959 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.719063 21948 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.721103 20845 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.722208 21948 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.722898 21959 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.724126 21959 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.724503 21959 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.727891 21032 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:29.729103 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:29.729522 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.730132 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.730710 21986 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.730984 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.731689 20845 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.732339 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.732878 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.727807 21957 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.735563 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.737701 20845 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:29.744669 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.732005 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.776194 21024 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240506 15:14:29.758250 21939 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.773399 21986 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.733493 21959 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.782562 21939 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.783396 21959 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.786852 21957 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.774936 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:29.787942 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.792094 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.794801 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.789970 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.796833 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.757634 21960 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:29.798095 21960 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.798473 21960 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.798964 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.800477 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.802589 20845 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:29.803644 21959 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:29.804040 21959 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.804427 21959 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:29.804746 21025 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:29.806486 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.797613 21957 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.807695 21957 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.808069 21957 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.808684 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.809010 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:29.810324 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.810564 21686 raft_consensus.cc:1226] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.813292 21920 consensus_queue.cc:1046] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:29.814544 21920 raft_consensus.cc:3032] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:29.814939 21920 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:29.801342 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.818065 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.817448 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.819505 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.815901 21920 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:29.821671 21959 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.822647 21920 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.811417 21939 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:29.823884 21939 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:29.824416 21939 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:29.828126 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.829515 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.831589 21986 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:29.832787 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.833657 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.833412 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.833977 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.834977 20951 raft_consensus.cc:2396] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240506 15:14:29.835100 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.836560 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.836874 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.838164 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:29.838649 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.839026 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.841511 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.844640 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.846853 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.848393 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.850208 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.851469 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.852391 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.852511 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.853546 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.856328 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.862089 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.862840 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.863842 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.866863 21959 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.867551 21957 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.868700 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.873329 21957 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.875134 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.875422 21960 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.880723 21960 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.895399 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:29.894845 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.900705 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.902153 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.911788 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.896781 21939 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:29.913400 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:29.913897 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 100989us
I20240506 15:14:29.914948 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.915409 21939 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:29.915702 20883 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:29.918742 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.920500 20883 raft_consensus.cc:1226] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:29.921763 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.922554 20883 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:29.924316 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.924357 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.926259 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.941421 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.943137 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.953135 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.954080 20879 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:29.955284 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.950227 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:29.957619 21970 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:29.958272 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.958981 20879 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:29.960486 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.961501 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:29.962302 21970 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:29.962837 21970 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:29.964138 22001 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:29.964184 21970 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:29.965124 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 116498us
I20240506 15:14:29.962296 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.968008 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:29.969184 21956 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:29.977855 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.981575 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:29.985633 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.002362 20924 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.002629 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.003279 22001 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.004396 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.004554 21960 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.005254 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.005633 20924 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.006134 21971 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.006557 21971 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.006955 21971 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.005105 21960 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.009560 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.012516 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.008704 22001 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.035985 21999 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:30.037981 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.044245 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.041523 21999 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.057801 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.059126 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.059490 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.061843 21960 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:30.063244 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:30.064849 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.054821 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.050704 21999 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.068925 20998 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.070617 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.072620 21974 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:30.074038 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 62305.9us
I20240506 15:14:30.074709 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.078583 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.079128 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.080027 20998 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.082145 20998 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.082644 21030 raft_consensus.cc:1226] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.083874 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.083811 21960 consensus_queue.cc:1046] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:30.087203 20998 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.087642 20845 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.085019 21960 raft_consensus.cc:3032] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:30.088116 20878 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:30.080790 20778 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:30.088420 21960 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:30.088523 21979 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.096704 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.100246 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.107321 22001 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.109077 21920 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.096840 21986 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.096475 21025 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.112463 21986 consensus_queue.cc:1046] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.113637 21957 raft_consensus.cc:3032] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.114002 21957 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:30.114734 21957 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.085490 21974 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.085280 21982 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.122344 20883 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:30.127804 21024 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.127966 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.107996 22001 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.132695 21656 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.132412 22001 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.135758 21984 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.132756 21959 consensus_queue.cc:1046] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240506 15:14:30.136554 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:30.137356 21959 raft_consensus.cc:3032] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.096520 21960 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.137698 21959 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:30.138252 21960 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.138479 21959 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.139688 21959 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.127802 21939 consensus_queue.cc:1046] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240506 15:14:30.141814 21939 raft_consensus.cc:3032] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.142180 21939 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:30.142928 21939 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.148098 21656 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.149453 21656 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.129242 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.130115 21957 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.152536 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.154124 20845 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.159512 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.163619 20998 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.131901 21982 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.144088 21939 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.167549 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:30.161599 20846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.130:40545 timed out after 1.533s (SENT) [suppressed 4 similar messages]
W20240506 15:14:30.176173 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 101526us
I20240506 15:14:30.158129 21984 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.162277 21951 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.190215 20875 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:30.170368 21974 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.210534 21971 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.166018 22007 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.162449 22008 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.180881 21972 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.184372 21979 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.207036 21999 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:30.257522 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 90068.4us
W20240506 15:14:30.259289 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 82400.4us
I20240506 15:14:30.243528 22008 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:30.260923 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.261518 20998 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.264602 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.269184 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.271135 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.272540 20998 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.273859 20998 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.240125 22001 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.277207 21982 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.277377 21920 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.289125 21920 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:30.289220 21982 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:30.260635 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 111242us
I20240506 15:14:30.277125 22007 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.265566 22008 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.275878 21979 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.275949 21972 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.298286 21656 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.290879 21920 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:30.303817 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 142243us
I20240506 15:14:30.307293 21984 raft_consensus.cc:686] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:30.308705 22005 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:30.329638 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 54559.1us
I20240506 15:14:30.316817 22011 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election won for term 2
W20240506 15:14:30.332465 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 71605.5us
I20240506 15:14:30.333734 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.334471 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:30.308545 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 80206.5us
I20240506 15:14:30.338723 21971 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.333348 22001 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.339372 21974 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.342097 21982 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.361709 22005 consensus_queue.cc:1046] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.362182 20924 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.363603 22005 raft_consensus.cc:3032] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.363693 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.356740 22018 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.364831 22001 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.364948 22018 raft_consensus.cc:686] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:30.365240 22001 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.365669 22001 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.365936 22018 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.360086 21970 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.358922 21999 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.356034 21984 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.375027 22007 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.375591 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.376773 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.385510 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.392138 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.393854 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.395227 20924 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.364020 22005 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.403482 20924 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.408866 22009 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:30.396939 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 98737.8us
I20240506 15:14:30.409548 22009 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.410173 22009 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.364020 22008 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.419680 22008 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:30.384985 21999 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.384790 21970 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.427736 22014 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.391440 22013 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.403643 22010 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:30.435163 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 99197.3us
I20240506 15:14:30.436758 20998 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.440124 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.444062 20949 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:30.446750 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.442431 22021 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.447551 21972 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.450485 21982 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:30.451603 21972 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:30.451936 21972 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:30.452741 21972 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.454231 21972 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.457165 22001 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.458642 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:30.459268 21979 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.462155 22019 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.439163 22012 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.463953 22011 raft_consensus.cc:686] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:30.440227 21920 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.473942 22011 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.475183 21920 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:30.425328 22005 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.476703 22005 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.472441 22001 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.479398 20778 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 2 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:30.491055 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:30.493934 20998 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.455652 22009 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.499837 22020 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.511457 22024 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.519407 22024 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.521703 22024 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:30.523100 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.527325 22009 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.536641 22024 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.537434 22024 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.539016 22024 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:30.539898 22020 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.540745 22024 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.540575 22020 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:30.541391 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 47531us
I20240506 15:14:30.541343 22024 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.542173 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.542614 22020 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:30.542853 21972 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.543684 22024 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:30.539842 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 136464us
I20240506 15:14:30.543934 20998 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.544145 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.544744 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.544099 20778 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 1 to 2, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 2 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:30.545536 22030 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.545584 22024 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.545100 20998 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.546703 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.547212 20924 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.548000 22030 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.548096 21972 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.548470 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.550179 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.550474 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.551095 20883 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:30.552626 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.552584 21654 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.554003 22011 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.554139 21654 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.559888 22018 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.557587 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.561439 21654 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.562230 22018 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.562625 21654 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.563515 22018 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.563928 21654 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:30.565089 22018 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.565449 22018 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:30.545830 20778 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 2 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:30.565805 22018 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.572947 21957 consensus_queue.cc:1046] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:30.574209 21939 raft_consensus.cc:3032] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.574611 21939 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:30.575372 21939 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.579380 21939 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.580143 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.580163 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.581212 20955 raft_consensus.cc:2396] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240506 15:14:30.583122 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.583200 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.584218 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.584817 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.585317 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.585635 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.585954 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.586561 20951 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:30.587054 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.587764 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.588435 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.588430 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.589447 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.590234 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.590912 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.595999 20878 raft_consensus.cc:2379] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:30.595844 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.589393 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.597002 20883 raft_consensus.cc:2379] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:30.597098 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.592517 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.594390 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.594946 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.595022 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.593786 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.597667 20949 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:30.600713 21959 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.600538 21957 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.601727 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.602648 21957 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.602763 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.604861 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.605578 20878 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:30.606021 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.606282 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.606815 20875 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:30.607015 20883 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:30.611681 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.623471 21654 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.608623 22018 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.613533 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.611544 20955 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.619845 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.613204 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.643635 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.609443 22024 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.649964 22024 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.650879 20846 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.652493 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.652750 22024 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:30.643826 22018 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:30.639762 22026 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.653908 21959 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.653945 20950 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.649422 21957 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.641865 22031 consensus_queue.cc:1046] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240506 15:14:30.644722 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.646212 20949 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.624234 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.656519 22020 raft_consensus.cc:3032] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.656883 22020 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:30.657042 22024 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.658180 21026 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.658283 22007 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.658663 22007 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:30.659037 22020 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.659401 22007 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.660604 22007 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.655891 22031 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:30.662612 22031 raft_consensus.cc:3032] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.663018 22031 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:30.664497 22031 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.665903 22031 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.667264 22020 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.668442 22029 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:30.669435 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45302.8us
I20240506 15:14:30.670123 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.672780 22029 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:30.673161 22029 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:30.673519 22029 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.679137 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.640244 22011 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.681067 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.653208 21939 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.682772 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.646617 20951 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.684567 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.687243 21959 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.688153 20924 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.686915 22029 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.688153 20846 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.689543 22030 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.690476 22029 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.690896 21959 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.691998 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.693876 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:30.693784 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.694761 20951 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.695338 21959 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.695775 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.696484 21959 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.696908 20846 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.697625 21959 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.698048 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.699226 20846 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.699383 21959 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.694269 22007 consensus_queue.cc:1046] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:30.700886 22007 raft_consensus.cc:3032] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.701187 22007 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:30.707919 20951 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:30.708343 21024 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:30.702018 22007 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.709811 21654 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:30.712883 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.714371 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.714548 22011 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.715096 22011 raft_consensus.cc:686] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:30.715931 22011 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.719508 21024 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:30.708487 21031 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:30.710768 21957 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.710548 22007 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.723259 21025 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:30.724522 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:30.711086 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.723965 21024 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:30.725605 21957 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.727423 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.728205 21957 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.728613 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.730360 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:30.730173 22029 consensus_queue.cc:1046] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.731477 22030 raft_consensus.cc:3032] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.731837 22030 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.732692 22030 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.733950 22030 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.736549 21957 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.737013 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.737834 21957 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.738240 20846 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.738986 21957 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.739394 20846 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.740445 21957 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.740839 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.741573 21957 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.741912 20846 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.742977 20846 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.743561 20776 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 2 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:30.751736 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:30.753101 22029 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.753741 22029 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.754580 22029 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.765504 21959 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.766288 21959 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.766870 20921 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:30.768020 22029 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.768735 22029 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.765884 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.770233 22029 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.771957 21957 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.772387 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.773532 20846 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.774336 21957 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.775152 21957 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.776012 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.773523 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:30.777294 22005 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:30.777793 22005 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.778712 22005 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.781221 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.782071 21959 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.782481 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.786881 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:30.789434 22007 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.790266 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.786881 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.793138 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.788882 21959 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.788340 22030 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:30.802492 22030 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.787756 20776 catalog_manager.cc:5484] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:30.794565 22020 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.795733 21939 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.793466 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.809667 21939 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.810997 20846 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.811729 21939 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.798085 20998 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.812695 22020 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.812811 20846 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.813597 21939 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.800662 20883 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.794184 22007 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.814715 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.816532 21939 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:30.817535 22011 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:30.819835 22030 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.820684 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.821440 21939 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.823347 22020 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.827109 20998 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.828227 22007 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.828111 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.829540 21939 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.830827 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:30.832477 21939 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:30.832983 21939 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:30.833434 21939 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:30.835294 22011 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:30.835325 22007 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.835975 22011 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:30.836727 20998 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.838348 22024 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.839175 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:30.839771 21031 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:30.841112 20846 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.845628 20881 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:30.846263 20874 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.846582 20877 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.847364 20880 raft_consensus.cc:1226] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.847702 21959 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.848625 20876 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.849675 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.849558 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.851337 20877 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.852025 20881 raft_consensus.cc:1226] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:30.854876 20881 raft_consensus.cc:1226] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.847376 20874 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.852214 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.852984 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.855782 20998 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:30.857378 20879 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.859230 20878 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:30.850769 22011 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:30.858302 21982 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.875141 20882 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.859473 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:30.866781 22009 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240506 15:14:30.885673 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.886988 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.888288 20880 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.885578 20877 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.889299 20880 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.890007 20877 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.890251 20880 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:30.891810 20881 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:30.875166 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:30.892627 20880 raft_consensus.cc:1226] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.893509 20877 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:30.892716 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:30.881997 22024 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:30.900472 20878 raft_consensus.cc:1226] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.900609 20875 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:30.904142 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.904711 20876 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.895815 22026 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:30.909750 21024 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.915367 22026 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:30.915777 22026 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.910801 21025 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.920464 21971 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.921497 22015 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:30.921159 21971 consensus_queue.cc:1046] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.922724 22015 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:30.923359 21984 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:30.923702 22019 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.923954 22015 raft_consensus.cc:3032] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.924156 22001 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.924808 22015 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.924875 22001 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.925982 22001 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.921422 22029 consensus_queue.cc:1046] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:30.924795 21984 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240506 15:14:30.927316 22001 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.927857 22029 raft_consensus.cc:3032] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.922762 21971 consensus_queue.cc:1046] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:30.928411 22029 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.928586 22005 consensus_queue.cc:1046] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.929065 21971 raft_consensus.cc:3032] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:30.929780 21956 raft_consensus.cc:3032] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:30.930234 21956 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.930001 22029 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.924135 22019 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.931135 21956 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.931358 22019 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.923537 22021 consensus_queue.cc:1046] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.932466 21956 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.932747 21999 raft_consensus.cc:3032] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:30.933161 21999 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.933903 21999 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.935257 21999 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.936667 21971 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:30.938400 21971 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.940171 21971 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:30.925844 22015 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:30.941906 22015 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.942942 20879 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.944725 20951 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.946231 22019 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.912830 20881 raft_consensus.cc:1226] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:30.930513 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:30.951079 22005 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:30.954468 21025 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.960008 22011 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:30.972107 20881 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.973460 20878 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:30.976884 21970 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.977982 22011 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:30.978334 21984 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:30.983680 22021 consensus_queue.cc:1046] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:30.931301 22029 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:30.994141 21972 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:31.010720 21939 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.028482 22030 consensus_queue.cc:1046] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240506 15:14:31.036985 22021 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240506 15:14:31.028215 21972 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.075320 21939 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:31.084061 22005 raft_consensus.cc:3032] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.084530 22005 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.084607 21688 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:31.085345 22005 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:31.091979 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80239.1us
I20240506 15:14:31.101469 22009 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.122094 22009 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.122956 22009 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.123843 22026 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.126085 22026 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:31.139393 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 66137.7us
I20240506 15:14:31.096711 21971 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240506 15:14:31.129720 22005 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.187546 20957 raft_consensus.cc:1226] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:31.131330 22009 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:31.188952 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 52738.8us
I20240506 15:14:31.102988 21959 consensus_queue.cc:1046] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20240506 15:14:31.184458 22027 log.cc:924] Time spent T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.055s user 0.000s sys 0.007s
I20240506 15:14:31.185431 21956 raft_consensus.cc:3032] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.200728 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:31.202852 21999 consensus_queue.cc:1046] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:31.129518 22015 consensus_queue.cc:1046] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20240506 15:14:31.188866 22019 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:31.218520 21972 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:31.183969 21970 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.220639 21956 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.245669 21024 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:31.241369 21970 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:31.199767 22038 log.cc:924] Time spent T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.070s user 0.007s sys 0.000s
I20240506 15:14:31.209038 21959 consensus_queue.cc:1225] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240506 15:14:31.232352 21688 raft_consensus.cc:2379] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:31.232105 22021 consensus_queue.cc:1046] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:31.232379 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:31.223178 22024 consensus_queue.cc:1046] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240506 15:14:31.263844 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:31.278540 22045 raft_consensus.cc:3032] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.278208 20878 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:31.279631 22030 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.280062 22030 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.280930 22030 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.282341 22030 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:31.284723 22042 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.097s user 0.002s sys 0.002s
I20240506 15:14:31.278976 22045 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:31.279107 22024 raft_consensus.cc:3032] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.286868 21970 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.287564 22024 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:31.287951 22045 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.288457 22024 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.288213 20776 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:31.289224 22045 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.240689 22004 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
W20240506 15:14:31.298806 22028 log.cc:924] Time spent T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240506 15:14:31.241716 22019 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.314465 22024 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.317891 22029 raft_consensus.cc:3032] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:31.318324 22029 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.319120 22029 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.298485 22004 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.320899 21956 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.321483 22004 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.322330 21956 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.322763 22004 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:31.333055 20951 raft_consensus.cc:1799] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Handling vote request from an unknown peer 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:31.334118 20951 raft_consensus.cc:2396] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240506 15:14:31.348702 20776 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 2 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:31.350872 20776 catalog_manager.cc:5484] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:31.333161 22019 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:31.396049 22044 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:31.396806 22044 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:31.419482 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 228535us
I20240506 15:14:31.424351 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:31.424896 22019 consensus_queue.cc:1046] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:31.425151 21026 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:31.425957 22019 raft_consensus.cc:3032] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.426327 22019 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.426590 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:31.427350 21690 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:31.424189 22005 consensus_queue.cc:1046] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240506 15:14:31.428723 22004 raft_consensus.cc:3032] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
W20240506 15:14:31.415055 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 195273us
I20240506 15:14:31.429169 22004 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.433926 22004 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.427085 22019 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.440411 22005 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240506 15:14:31.443215 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:31.443908 21686 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:31.444368 20996 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:31.446204 21972 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:31.446560 22019 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.435338 22044 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:31.435302 22004 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.449561 22005 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:31.449973 22005 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.450883 22005 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:31.460939 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 222079us
I20240506 15:14:31.462453 20878 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:31.464151 21024 raft_consensus.cc:1226] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:31.471132 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:31.478235 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 47267.9us
I20240506 15:14:31.477649 20922 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:31.484840 21970 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.488442 21026 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:31.488602 22021 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:31.489408 22021 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:31.491057 22001 consensus_queue.cc:1046] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:31.492090 22001 raft_consensus.cc:3032] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.492502 22001 raft_consensus.cc:729] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.493007 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:31.493381 22001 consensus_queue.cc:260] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.494998 22001 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.500133 22046 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:31.500612 22046 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:31.500939 22046 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:31.503525 22005 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:31.509387 22046 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.511590 22046 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:31.521081 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:31.521386 22062 consensus_queue.cc:1046] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:31.521857 21686 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:31.523339 22055 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:31.526719 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:31.526849 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45126.2us
I20240506 15:14:31.502386 22029 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:31.529213 22011 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:31.502813 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:31.530530 22055 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:31.551033 22055 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.536094 22046 consensus_queue.cc:1046] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:31.538110 22060 consensus_queue.cc:1046] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240506 15:14:31.541036 21956 raft_consensus.cc:3032] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.533612 22062 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 15:14:31.562602 20881 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:31.563091 21956 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.569149 21956 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.570670 20879 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:31.571687 21025 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:31.574200 22046 raft_consensus.cc:3032] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.574611 22046 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.575429 22046 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.576704 20845 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:31.581961 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:31.602013 20874 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:31.599004 20881 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:31.624845 21686 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:31.622720 21959 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:31.627043 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:31.627861 20876 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:31.630303 20922 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:31.634832 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 72712us
I20240506 15:14:31.601189 22005 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:31.607581 22045 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:31.619028 21956 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.629127 22046 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.647544 21030 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:31.650568 21025 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:31.651094 22001 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:31.651595 22001 raft_consensus.cc:686] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.641014 20879 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:31.642695 22045 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:31.687500 20876 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:31.634496 21959 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.664038 22044 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:31.642385 22005 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.674332 22045 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:31.704450 21027 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:31.674412 22015 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240506 15:14:31.706152 21999 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:31.713163 22059 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240506 15:14:31.674189 22058 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240506 15:14:31.751259 21690 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:31.700970 22044 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:31.746428 21686 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:31.689831 22001 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:31.743561 21999 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.746625 22059 raft_consensus.cc:3032] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:31.768388 21690 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:31.769783 20953 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:31.775486 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:31.775686 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:31.780779 20950 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:31.783541 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:31.767508 22059 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:31.784634 22009 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20240506 15:14:31.774246 22019 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240506 15:14:31.803681 21024 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240506 15:14:31.948493 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 127512us
I20240506 15:14:31.804867 22055 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.148s
I20240506 15:14:31.817976 21959 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:31.836887 22005 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:32.036228 22067 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20240506 15:14:32.038033 22061 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.251s
I20240506 15:14:31.795938 22062 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:32.041908 20953 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:14:32.027281 21972 consensus_meta.cc:287] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.581s user 0.008s sys 0.010s
I20240506 15:14:31.795424 22060 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.470s
I20240506 15:14:32.492111 22019 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.492966 22019 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.495671 22019 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:32.495795 22060 consensus_queue.cc:1225] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20240506 15:14:32.496590 22066 log.cc:924] Time spent T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20240506 15:14:32.496979 22069 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:32.281950 21972 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:32.622048 22069 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:32.485224 22044 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:32.794466 21031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.036s sys 0.114s
W20240506 15:14:32.802572 21031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s user 0.036s sys 0.114s
I20240506 15:14:32.526851 22005 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.541014 22060 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.459487 22071 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.361s
I20240506 15:14:32.822038 22062 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:32.623333 22069 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.839238 22060 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.838732 21956 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.864953 21956 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.867524 21956 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:32.834452 22005 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.863018 22060 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:32.879333 22046 consensus_meta.cc:287] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 1.226s user 0.007s sys 0.005s
I20240506 15:14:32.902837 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1138) took 1097 ms. Trace:
W20240506 15:14:32.903118 20998 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.029s sys 0.035s
W20240506 15:14:32.903690 20998 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.029s sys 0.035s
I20240506 15:14:32.876494 22005 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:32.903432 21031 rpcz_store.cc:276] 0506 15:14:31.805519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.805797 (+ 278us) service_pool.cc:225] Handling call
0506 15:14:31.806207 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:31.806635 (+ 428us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:31.806674 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:31.812168 (+ 5494us) log.cc:842] Serialized 26 byte log entry
0506 15:14:32.803090 (+990922us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:32.803134 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:32.803210 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:32.902555 (+ 99345us) raft_consensus.cc:1671] finished
0506 15:14:32.902581 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:32.902746 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":538,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9312,"thread_start_us":796957,"threads_started":1,"wal-append.queue_time_us":1049175}
I20240506 15:14:32.918007 21982 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:32.923828 21982 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.927554 21982 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:32.939968 22077 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.941313 22077 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:32.946568 21956 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.948575 21956 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.955118 22082 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.956020 22082 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.958890 22082 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:32.963140 22045 consensus_meta.cc:287] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 1.254s user 0.008s sys 0.003s
I20240506 15:14:32.963102 22080 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:32.963658 22045 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.966749 22045 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:32.963806 22080 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:32.997954 22080 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:32.806659 20996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.022s sys 0.036s
W20240506 15:14:33.013314 20996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.017s user 0.022s sys 0.036s
I20240506 15:14:33.015911 22074 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.016741 22074 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.019610 22074 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.020233 22074 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.021081 22074 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.023954 22074 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.028215 22077 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.029320 22077 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.033984 21956 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:33.034960 20921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.011s sys 0.002s
W20240506 15:14:33.035333 20921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.011s sys 0.002s
I20240506 15:14:33.036499 22082 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.037060 22082 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.039973 22082 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.040616 22082 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.041143 22082 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:33.047330 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.005s sys 0.007s
W20240506 15:14:33.047813 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.005s sys 0.007s
I20240506 15:14:33.049710 22079 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.050926 22079 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.054042 22074 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.054713 22074 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:33.056375 22001 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.010s sys 0.000s
W20240506 15:14:33.056819 22001 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.010s sys 0.000s
W20240506 15:14:33.061688 20997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.001s sys 0.008s
W20240506 15:14:33.062156 20997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.001s sys 0.008s
W20240506 15:14:33.062700 21027 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.306s user 0.004s sys 0.010s
W20240506 15:14:33.063081 21027 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.312s user 0.004s sys 0.010s
W20240506 15:14:33.064496 20924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.006s sys 0.007s
W20240506 15:14:33.064939 20924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.006s sys 0.007s
I20240506 15:14:33.091575 22084 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:33.092375 22084 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.096148 22084 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.099799 22084 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.100644 22084 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.103042 22084 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.111873 22078 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.112581 22078 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.115816 22078 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.125886 21972 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.126603 21972 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.129530 21972 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.130201 21972 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.130895 21972 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.133826 21972 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.138427 22078 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.139124 22078 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.142207 22078 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.143152 21972 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.143779 21972 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.148312 22084 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.149277 22084 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.166556 22088 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.167671 22088 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.029911 22077 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.044085 22082 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.142791 22078 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.251323 22095 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.053570 22079 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.244781 21972 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.249915 22096 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.302620 22087 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.307189 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1135) took 1557 ms. Trace:
I20240506 15:14:33.303725 22087 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.366251 22088 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.366384 22087 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.364090 22097 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.361243 22096 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.367640 22097 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.369537 22096 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.352723 22095 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.371177 22097 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.371601 22095 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.362736 22077 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.301486 22078 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.352906 21972 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.514246 21972 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.516741 22001 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.517410 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 21) took 1477 ms. Trace:
I20240506 15:14:33.517969 22001 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.520699 22005 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.521893 22005 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.524691 22046 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.525696 22046 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.529073 22062 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.530299 22062 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.532822 22080 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.533914 22080 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.530690 22096 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.516731 22095 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.537271 22099 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.538203 22095 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:33.540443 20923 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.005s sys 0.005s
I20240506 15:14:33.307767 21027 rpcz_store.cc:276] 0506 15:14:31.750259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.750415 (+ 156us) service_pool.cc:225] Handling call
0506 15:14:31.750815 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:33.063578 (+1312763us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:33.063625 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:33.122104 (+ 58479us) log.cc:842] Serialized 26 byte log entry
0506 15:14:33.306707 (+184603us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:33.306760 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:33.306854 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:33.306916 (+ 62us) raft_consensus.cc:1671] finished
0506 15:14:33.306934 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:33.307113 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":32767,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3034,"thread_start_us":1496178,"threads_started":2,"wal-append.queue_time_us":124826}
I20240506 15:14:33.550370 22083 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.553278 22079 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.553133 22086 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.567420 22098 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.581241 22079 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:33.583714 20923 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.006s sys 0.005s
I20240506 15:14:33.547878 22082 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.593112 22096 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.602587 22082 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.605222 22082 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.605846 22082 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.606540 22082 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.608774 22083 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.518178 20955 rpcz_store.cc:276] 0506 15:14:32.040216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:32.046961 (+ 6745us) service_pool.cc:225] Handling call
0506 15:14:32.487154 (+440193us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:32.803100 (+315946us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:32.817682 (+ 14582us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:32.873414 (+ 55732us) log.cc:842] Serialized 26 byte log entry
0506 15:14:33.181695 (+308281us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:14:33.181761 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:33.181864 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:33.483828 (+301964us) raft_consensus.cc:1671] finished
0506 15:14:33.483917 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:33.517104 (+ 33187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":301565,"raft.queue_time_us":229936,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"thread_start_us":307459,"threads_started":1,"wal-append.queue_time_us":299739}
I20240506 15:14:33.595964 22096 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.622778 21956 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.623661 21956 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.616851 22098 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.636132 22079 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.619405 22046 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.651453 22079 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:33.651262 22029 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.629770 21956 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.630857 22096 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.659184 22083 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.658783 22029 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.653510 22046 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.660331 22082 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.654016 22079 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.673846 22029 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.674115 22104 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.674666 22062 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.674456 22046 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.636036 22086 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.677572 22104 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.685206 22104 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.671756 22005 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.686677 22001 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.695783 22096 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.698273 22096 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.698107 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.698966 22096 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.700218 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.701189 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.702051 20949 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 1.
I20240506 15:14:33.703653 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.704676 22079 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.705302 22079 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.706060 22079 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.699740 20953 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:33.709656 22046 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.687355 21956 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.687873 22080 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.717042 21956 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.717340 21654 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:33.719727 21956 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.719988 22109 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.719388 21654 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:33.688572 21972 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.720121 22053 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:33.721181 21972 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.653076 22099 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.721596 22053 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.724313 22053 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:33.724040 21972 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.726540 21972 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.726761 22062 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.727110 21972 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.728435 22104 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.728226 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:33.728478 21688 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240506 15:14:33.720865 22109 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.727617 22062 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.699823 22096 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.730417 22104 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.735500 22029 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.736356 22029 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.737785 22104 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.738060 22031 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.694836 22082 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.738924 22029 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.739318 22082 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.741969 22082 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.743599 21687 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:33.745951 22086 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.732715 22080 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.757994 22098 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.744295 21956 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.769222 21956 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.733409 22108 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.732573 22001 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.773479 22108 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.793828 22099 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.795034 22031 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.774629 21956 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.793864 20951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:35794 (request call id 361) took 2114 ms. Trace:
I20240506 15:14:33.802965 22078 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.803632 22001 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.797722 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 40) took 2143 ms. Trace:
I20240506 15:14:33.810719 22095 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.810482 21030 rpcz_store.cc:276] 0506 15:14:31.653616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.653981 (+ 365us) service_pool.cc:225] Handling call
0506 15:14:33.796147 (+2142166us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:33.796649 (+ 502us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:33.796986 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:33.797026 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:33.797103 (+ 77us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:33.797195 (+ 92us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:33.797276 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:33.797468 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19488128}
I20240506 15:14:33.811794 22001 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.804631 22080 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.814246 22080 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.814980 22089 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:33.815460 22089 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:33.815822 22089 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:33.806560 22096 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.808236 20951 rpcz_store.cc:276] 0506 15:14:31.679145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.693618 (+ 14473us) service_pool.cc:225] Handling call
0506 15:14:33.787126 (+2093508us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540038de54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e274d30 00007f630e2f817a 00007f6313235a0e 00007f630c928817 00007f630c9287a2 00007f630c9286d1 00007f630c92864c 00007f630c927902 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:33.793516 (+ 6390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2951550336}
I20240506 15:14:33.809484 22109 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.752764 22029 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.818177 22108 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.751601 21972 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.836323 22015 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.837338 22015 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.830001 22029 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.849694 22029 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.831537 22031 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.828599 22098 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.857336 22109 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.858566 22108 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.858850 22098 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.859218 22108 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.854419 22080 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.860425 22095 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.860635 22080 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.859134 22109 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.863615 22096 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.864373 22096 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.858580 22107 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.861197 22095 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.861584 22108 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.862293 22105 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.866509 22096 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.846755 21956 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.854707 22001 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.868395 22062 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.876960 22098 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.878372 22109 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.879930 22106 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.875016 22105 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.843909 22053 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.887668 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.888753 21031 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:33.888435 22001 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.891386 22001 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.892573 22073 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.893282 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:33.893935 22007 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.893474 22073 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.894675 22007 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.898015 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.898339 22073 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.898991 21026 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:33.899619 22108 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.900445 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.900579 22108 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.901429 21025 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:33.876672 22031 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:33.866235 22107 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.880798 22080 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.910849 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.911883 21032 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:33.915413 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.916224 20954 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:33.909106 22099 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.916646 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.915838 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.917340 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.917306 22099 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.918015 20953 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:33.918308 20949 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:33.919607 22099 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.921038 22031 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.922986 22007 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.879168 22086 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.924115 22086 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.926591 22086 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.926663 21970 raft_consensus.cc:3032] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:33.927345 21970 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:33.917474 20957 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:33.882268 22062 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.928872 22062 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.892028 22024 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:33.933143 22024 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:33.935856 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.936792 21030 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:33.938359 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:33.931802 22062 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.940924 20951 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:33.948474 22095 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.948751 21972 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.923418 22031 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.951056 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.935936 22024 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:33.954748 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.946699 22096 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:33.956799 22015 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.958714 21029 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:33.955483 21031 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:33.950032 21956 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.946516 22007 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:33.952062 22113 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:33.962472 22107 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.946481 22106 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.964749 21970 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.953888 22108 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.969516 22105 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.967814 22096 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:33.971177 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.974814 21029 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:33.977669 22113 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:33.984903 20846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.208s user 0.058s sys 0.093s
I20240506 15:14:33.993057 22096 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:33.993803 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.994596 21029 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:33.995615 22053 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.995725 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 35) took 2452 ms. Trace:
I20240506 15:14:33.996126 20878 rpcz_store.cc:276] 0506 15:14:31.536019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.536165 (+ 146us) service_pool.cc:225] Handling call
0506 15:14:31.536491 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:31.536797 (+ 306us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:31.536827 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:31.537266 (+ 439us) log.cc:842] Serialized 26 byte log entry
0506 15:14:31.759968 (+222702us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:14:31.760005 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:31.760078 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:33.987922 (+2227844us) raft_consensus.cc:1671] finished
0506 15:14:33.987967 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:33.988168 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1018,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":222338,"threads_started":1,"wal-append.queue_time_us":261952,"wal-append.run_cpu_time_us":3183,"wal-append.run_wall_time_us":1827440}
I20240506 15:14:33.984403 21956 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:33.998356 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.004688 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.005627 20952 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.005924 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:34.006692 20846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.230s user 0.059s sys 0.094s
I20240506 15:14:34.007145 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:34.007313 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 2.2306e+06us
I20240506 15:14:34.005627 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.009059 20949 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.009450 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.010128 21027 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:34.011646 21029 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240506 15:14:34.012723 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:33.976747 22007 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.012902 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.013392 21030 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.014357 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.014463 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.014830 22007 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.015094 21027 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.015340 21032 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.053179 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.053808 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.054329 20951 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:34.054646 21025 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.056154 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.057014 21032 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.058295 22106 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.116432 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:34.061196 20879 consensus_meta.cc:287] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 2.383s user 0.013s sys 0.006s
I20240506 15:14:34.121534 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.125762 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.060528 22059 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.129536 20879 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:34.129835 20951 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.130133 22059 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.130589 20879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1079) took 2550 ms. Trace:
I20240506 15:14:34.007351 22114 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.131330 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.132190 20949 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.133786 20879 rpcz_store.cc:276] 0506 15:14:31.580652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.603284 (+ 22632us) service_pool.cc:225] Handling call
0506 15:14:31.638514 (+ 35230us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:34.130188 (+2491674us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:34.130486 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:34.136729 21032 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.136974 22113 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.137687 22113 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.138476 22079 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.139017 22118 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.140004 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.138260 22113 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.152858 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.173215 20956 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.174795 21960 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.175419 21960 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.176241 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.176954 20949 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.177561 21960 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.143501 21025 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.178186 21960 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.139777 22118 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.178732 21960 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.179551 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.181492 20955 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.186278 21960 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.186548 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.188969 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.187695 21960 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.192009 21960 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.193516 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.194358 21960 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.194418 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.194931 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.195128 21960 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.196345 21960 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.197216 21999 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.198683 21999 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.212930 22113 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.198948 21960 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.194936 22055 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:34.218335 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 210008us
I20240506 15:14:34.183733 22114 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.274091 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.296042 21027 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.297359 22020 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.298154 22020 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.245091 22055 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.307276 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 173) took 3327 ms. Trace:
I20240506 15:14:34.318990 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.319638 20949 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:34.233158 22089 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.321295 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.322073 20955 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
W20240506 15:14:34.322971 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 66095.6us
I20240506 15:14:34.323554 22089 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.324129 20996 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.324357 22019 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:34.325345 20996 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.325919 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.326285 20996 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.326717 21031 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:34.326964 22106 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:34.327363 22096 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:34.316707 21960 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.327833 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.330672 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.272159 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.330529 22096 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:34.327311 20996 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.315574 22113 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.333392 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.316032 22119 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.327711 22007 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:34.342898 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.327464 22106 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:34.348047 20949 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:34.345703 21027 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.342948 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:34.344497 20996 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.340839 22046 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.350553 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.352610 20956 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.348075 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.350270 22020 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.348410 22007 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:34.353756 21024 raft_consensus.cc:2379] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:34.345305 22096 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.357468 22123 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.357403 21960 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.344776 21687 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:34.353947 22079 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240506 15:14:34.360518 21024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1139) took 2540 ms (client timeout 2994 ms). Trace:
I20240506 15:14:34.349220 22106 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.355599 22007 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.347337 22118 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:34.365417 21024 rpcz_store.cc:269] 0506 15:14:31.820083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.856469 (+ 36386us) service_pool.cc:225] Handling call
0506 15:14:34.301111 (+2444642us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:34.360443 (+ 59332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135093504}
I20240506 15:14:34.346149 21025 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.348891 22113 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.375674 21960 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.375885 22113 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.373118 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.376667 21031 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.367209 22079 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:34.377553 22079 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.377604 21653 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:34.377874 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.378548 21030 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.379388 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.358753 22105 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:34.362599 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.385736 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.386669 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.387014 22011 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:34.387984 22114 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.394780 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.395856 21031 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:34.358258 22123 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.318485 20776 rpcz_store.cc:276] 0506 15:14:30.979920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:30.980138 (+ 218us) service_pool.cc:225] Handling call
0506 15:14:34.307005 (+3326867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:31.773057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:32.040745 (+267688us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:32.046913 (+ 6168us) write_op.cc:435] Acquired schema lock
0506 15:14:32.046950 (+ 37us) tablet.cc:660] Decoding operations
0506 15:14:32.625540 (+578590us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:32.625641 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:32.625687 (+ 46us) tablet.cc:683] Acquiring locks for 3 operations
0506 15:14:32.626146 (+ 459us) tablet.cc:699] Row locks acquired
0506 15:14:32.626166 (+ 20us) write_op.cc:260] PREPARE: finished
0506 15:14:32.626359 (+ 193us) write_op.cc:270] Start()
0506 15:14:32.626497 (+ 138us) write_op.cc:276] Timestamp: P: 1715008472626299 usec, L: 0
0506 15:14:32.626522 (+ 25us) op_driver.cc:331] REPLICATION: starting
0506 15:14:32.856837 (+230315us) log.cc:842] Serialized 1316 byte log entry
0506 15:14:33.673283 (+816446us) op_driver.cc:444] REPLICATION: finished
0506 15:14:33.683646 (+ 10363us) write_op.cc:301] APPLY: starting
0506 15:14:33.684552 (+ 906us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:33.684840 (+ 288us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:33.685051 (+ 211us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:33.689560 (+ 4509us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:33.751390 (+ 61830us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0506 15:14:33.785336 (+ 33946us) write_op.cc:312] APPLY: finished
0506 15:14:33.819009 (+ 33673us) log.cc:842] Serialized 34 byte log entry
0506 15:14:33.819217 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:33.819403 (+ 186us) write_op.cc:454] Released schema lock
0506 15:14:33.819462 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:33.820038 (+ 576us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:33.820066 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":9898,"apply.run_cpu_time_us":4526,"apply.run_wall_time_us":136838,"num_ops":3,"prepare.queue_time_us":342147,"prepare.run_cpu_time_us":15983,"prepare.run_wall_time_us":1422531,"raft.queue_time_us":491859,"raft.run_cpu_time_us":3651,"raft.run_wall_time_us":18572,"replication_time_us":1046431,"thread_start_us":1028214,"threads_started":4,"wal-append.queue_time_us":320742}]]}
I20240506 15:14:34.402194 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.402437 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.405972 20949 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.403420 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.407166 21029 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.404448 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.410501 21027 raft_consensus.cc:2379] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:34.412568 22121 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.413264 22121 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.420296 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.421461 21028 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.423489 21028 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:34.425161 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.426046 21032 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.359189 22046 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.430259 22118 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.431090 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.431033 22118 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.431900 21023 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.433490 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.385001 22105 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:34.415932 22121 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.435420 22118 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.437332 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.430366 21960 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.455456 22124 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:34.456230 22124 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.439090 21024 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.439034 22084 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.439193 22113 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.442042 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.442268 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.442405 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.443679 22074 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.443616 22055 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.444902 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:34.445852 22053 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:34.446856 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.464707 20955 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:34.464496 22113 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.465174 20958 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.466210 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.466661 22113 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.466820 20958 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.467757 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:34.468412 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 149271us
I20240506 15:14:34.447438 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.452088 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.473863 20949 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.430368 22125 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.456037 21960 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.453568 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.483978 21031 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:34.441567 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.460153 20953 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.460351 20957 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.448837 22105 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.450850 22127 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.464715 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.485121 20956 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.461472 22118 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.483117 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:34.483350 21684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.009s sys 0.025s
I20240506 15:14:34.501657 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.401837 22119 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.508991 22130 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.509171 20950 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240506 15:14:34.509711 20950 pending_rounds.cc:76] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 4
I20240506 15:14:34.509672 22130 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.515635 22121 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.510222 20950 pending_rounds.cc:97] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240506 15:14:34.516388 22127 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.516741 20950 raft_consensus.cc:2866] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:34.518815 22127 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.522090 20955 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:34.524577 22128 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.525240 22128 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.527472 22128 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.530974 21960 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.531586 21960 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:34.538944 22118 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.540084 22123 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:34.505277 21684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s user 0.009s sys 0.026s
I20240506 15:14:34.540482 22103 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.512032 22130 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.559553 22124 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.541669 22128 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.515916 22007 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.564496 22128 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.567469 22103 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.575186 22119 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.575857 22128 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.544051 22121 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.583410 22121 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.546296 22125 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.576431 22007 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.579313 22106 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.862771 22106 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.863590 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.864111 21024 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:34.864464 21026 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.864638 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.865334 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:34.865918 21030 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.867775 21970 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.592326 22133 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.868515 22133 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.870410 22133 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.870471 20950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 26) took 1171 ms. Trace:
I20240506 15:14:34.593315 22121 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.871220 22121 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.873189 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54786 (request call id 190) took 3192 ms. Trace:
I20240506 15:14:34.873323 22121 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:34.597837 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 91199.8us
I20240506 15:14:34.870894 20950 rpcz_store.cc:276] 0506 15:14:33.698720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:33.705250 (+ 6530us) service_pool.cc:225] Handling call
0506 15:14:33.705884 (+ 634us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:34.518098 (+812214us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:34.870254 (+352156us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b540019fd54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:34.870415 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1843248896}
I20240506 15:14:34.875159 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.875566 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:34.875875 20950 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.594256 22131 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.876608 22131 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.876986 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.873669 20779 rpcz_store.cc:276] 0506 15:14:31.680861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:31.681005 (+ 144us) service_pool.cc:225] Handling call
0506 15:14:34.872969 (+3191964us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24714,"reported_tablets":3}
I20240506 15:14:34.878798 22131 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.879364 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.880448 22114 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:34.593808 22129 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.880144 20883 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.880998 20954 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.881270 22129 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.882230 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.882545 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.594503 22130 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.883190 20954 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.593492 22119 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:34.883944 20950 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.884038 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.602576 22118 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.609086 22135 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:34.884719 20954 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.607872 22079 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.883512 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.885578 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.886168 20883 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.886539 22079 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.612514 22125 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.608414 22096 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.886907 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.888692 22096 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.627739 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.886183 20954 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.889878 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.890002 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.629446 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.890520 21027 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:34.890935 20876 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:34.891583 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.891906 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.883371 22130 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.630151 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.892822 20876 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.892882 20874 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:34.893687 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.634796 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.893963 22130 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.633237 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:34.654878 22103 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.894315 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.894618 20880 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.894918 20877 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.895357 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.895489 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.895759 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.675173 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.683702 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.743494 20780 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 3 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:34.629827 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.752007 22074 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:34.775038 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.787245 20996 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.795202 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.795650 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.801343 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.808333 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.812988 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:34.813555 21690 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:34.896832 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.904596 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.881011 22114 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:34.898445 21025 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.813887 22071 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:34.896241 20882 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.823793 22046 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.828536 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.828387 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.902069 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.836628 21685 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:34.906956 21656 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:34.532171 21960 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.883829 22119 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.588811 22105 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.897399 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.890534 20954 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.885040 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:34.884941 22135 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.892275 21027 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:34.893739 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.650519 22069 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:34.750597 22126 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.239s user 0.004s sys 0.000s
I20240506 15:14:34.573029 21654 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:34.921607 20880 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.920203 20998 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:34.900401 22129 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.957368 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.957700 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:34.958269 20880 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.958541 22079 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:34.959023 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.959734 20880 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.959913 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:34.922228 21032 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:34.960623 22079 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:34.961215 22096 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:34.922627 22074 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.962414 22096 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:34.962872 22096 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.961319 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.963862 22074 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:34.922686 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 135508us
W20240506 15:14:34.964792 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 407528us
I20240506 15:14:34.964840 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.922838 21029 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:34.965581 21032 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:34.965610 21654 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.966562 22097 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:34.923032 20955 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.967459 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:34.924413 22103 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:34.968120 22103 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:34.965018 20996 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:34.968042 20880 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.924424 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.924571 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.969911 20996 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:34.969967 21025 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:34.970456 20882 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.968827 22103 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.970695 21971 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:34.971345 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.971165 21971 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:34.972095 20882 raft_consensus.cc:2379] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.972751 22097 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:34.973358 22097 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:34.973687 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.974457 21971 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.925252 21024 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:34.927064 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.975639 20949 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:34.931180 21031 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:34.932155 20953 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.933538 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.928733 21960 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.974212 22097 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.977948 20883 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.978451 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.978710 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.979028 20876 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.933912 20950 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.980078 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.980366 22020 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:34.931792 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.980852 20876 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.981184 20879 raft_consensus.cc:2379] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.981767 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:34.982057 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.982451 20876 raft_consensus.cc:2379] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.982793 20879 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:34.983187 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.983662 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.939568 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.984674 22053 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:34.984962 21023 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:34.940330 20874 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.939860 22007 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:34.986012 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.986248 22007 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:34.974288 20882 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.986645 22007 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:34.987097 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.987412 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.988075 20882 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.988152 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:34.947453 22046 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240506 15:14:34.989104 20874 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.989063 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.928712 22119 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:34.990021 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.990388 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.990845 20874 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.929225 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.939064 22135 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:34.922442 20956 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:34.920804 20877 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.923264 20952 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:34.930361 21656 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:34.979374 20883 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.953368 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.992308 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.994925 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.995144 20875 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.995721 20883 raft_consensus.cc:2379] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.925496 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.923569 22114 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:34.997107 20881 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.998131 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.998817 20881 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:34.999545 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.996155 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.980863 22020 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.000569 20875 raft_consensus.cc:2379] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.000106 20881 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:34.924726 20958 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:34.948398 21684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34002 (request call id 41) took 1387 ms. Trace:
I20240506 15:14:34.930320 22105 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:34.997043 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.002125 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:34.924490 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 240979us
I20240506 15:14:35.002645 20877 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:35.000770 22020 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.952915 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.003118 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.002748 20881 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:35.003594 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.004063 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.982102 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.007851 21031 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.004783 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.005390 20879 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:35.005647 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:34.992208 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:34.984982 22079 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.009574 22079 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.001780 21684 rpcz_store.cc:276] 0506 15:14:33.560729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:33.561324 (+ 595us) service_pool.cc:225] Handling call
0506 15:14:33.561949 (+ 625us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:33.563061 (+ 1112us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:33.563112 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:33.587737 (+ 24625us) log.cc:842] Serialized 26 byte log entry
0506 15:14:34.205963 (+618226us) spinlock_profiling.cc:243] Waited 6.28 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630b3a716e 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f630df704eb
0506 15:14:34.565020 (+359057us) raft_consensus.cc:1631] Marking committed up to 6
0506 15:14:34.749986 (+184966us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:34.757194 (+ 7208us) log.cc:842] Serialized 14 byte log entry
0506 15:14:34.757652 (+ 458us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:34.757731 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:34.948123 (+190392us) raft_consensus.cc:1671] finished
0506 15:14:34.948155 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:34.948315 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":184476,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":1699,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":405,"spinlock_wait_cycles":51955840,"thread_start_us":907193,"threads_started":2,"wal-append.queue_time_us":909614}
I20240506 15:14:35.009694 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.011196 21023 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.011386 20881 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.009986 22079 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.010443 20877 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.009035 21024 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.012626 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.012763 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.013283 20877 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:35.013517 20879 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.014189 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.014794 20877 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.014828 21027 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.015723 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.010756 22046 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.016441 22046 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.016492 22105 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.016916 22046 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.017691 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.001421 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.005923 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.018654 20996 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.016968 22105 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.019280 20883 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.019555 22105 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.020215 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.018553 22074 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.021538 22074 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.021698 20882 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.021957 22074 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.022156 22106 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.018707 20875 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.022611 22106 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.023240 22106 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:34.983899 20876 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.010676 20921 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.029865 20883 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.030681 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.028295 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.022572 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.023393 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.034973 20921 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.033260 21030 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:35.045609 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.045907 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.046541 20881 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.046669 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 86821.9us
I20240506 15:14:35.046788 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.047816 20883 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:35.045694 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.049309 20877 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.047734 20880 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:35.046335 20876 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:35.014500 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.060387 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.052716 20882 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:35.052664 20875 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.064831 20875 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.065753 20882 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:35.061204 20880 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:35.069015 20996 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.034479 21971 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.021129 22071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35794 (request call id 284) took 6153 ms. Trace:
I20240506 15:14:35.071353 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.072024 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.077066 20877 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:35.031556 22103 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.077584 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.078578 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:35.060799 22084 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.070996 21971 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.081339 20880 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:35.071421 22069 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.082404 20883 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:35.081640 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.033926 22079 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.046499 22096 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.089989 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.090651 20877 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:35.091421 22105 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.086318 22069 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.094475 22069 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.072517 20879 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:35.085804 21971 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.082871 22084 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.099120 22084 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.101164 22020 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.103312 22020 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.103839 22020 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.105329 22103 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.093598 22105 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.113925 20921 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.117693 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.118535 22074 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.119048 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.135483 21024 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.140225 20921 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.061759 22046 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.144193 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.146394 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.147492 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.120195 21970 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.119786 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.104210 22020 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.154563 20921 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.135840 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.142530 22105 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:35.150885 21026 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.097704 22007 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:35.161049 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 92097.1us
I20240506 15:14:35.149636 22103 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.166006 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.166182 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.166641 21030 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.151152 21970 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.168176 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.168861 20877 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.170696 20996 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.171525 20996 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.156550 22020 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.161760 22055 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:35.180697 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.181385 20998 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.159061 21025 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:35.172475 22004 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.183754 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.166303 22103 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.187158 21030 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.164860 21023 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:35.172814 22059 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.187901 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.207361 20877 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.200508 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.210781 20875 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240506 15:14:35.182776 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 252461us
I20240506 15:14:35.192420 22021 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.212213 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.166028 22100 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.212713 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:35.213090 20877 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.213444 22100 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.213912 22100 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.214190 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.215682 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.073863 22071 rpcz_store.cc:276] 0506 15:14:28.867791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:28.867915 (+ 124us) service_pool.cc:225] Handling call
0506 15:14:28.928325 (+ 60410us) log.cc:842] Serialized 501 byte log entry
0506 15:14:29.084551 (+156226us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b180015f100. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b33e2ea 00007f6314a2dfe5 00007f630b3c23d4 00007f630b3c1cc5 00007f630e269570 00007f630e2cf363 00007f630e2f5a32 00007f630e2ede6a 00007f6313232479
0506 15:14:35.021000 (+5936449us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86310,"raft.queue_time_us":91861,"raft.run_cpu_time_us":1330,"raft.run_wall_time_us":8358,"spinlock_wait_cycles":64109440,"thread_start_us":66115,"threads_started":1,"wal-append.queue_time_us":113574,"wal-append.run_cpu_time_us":1500,"wal-append.run_wall_time_us":1002135}
I20240506 15:14:35.173018 21970 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.193783 22079 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.197503 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.197994 21971 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.198532 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.199542 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.200492 22152 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.200917 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.233372 20880 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.173678 22105 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.235164 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.202643 22004 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.205472 22086 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:35.212503 22021 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.188803 22007 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.217201 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.213932 20921 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.247498 21024 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.220207 20881 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:35.217762 20877 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.247973 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.211931 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.165102 22069 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.249011 20921 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.249534 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.249799 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.250180 22069 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.250638 22068 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.250721 22069 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.251099 22068 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.251133 22069 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.251595 22068 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.252523 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.233034 22152 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.253676 22152 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.239981 22004 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.205993 22103 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.168081 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.243669 22021 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.256610 20879 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.257844 20879 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:35.259099 20879 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.208554 22096 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.207065 22059 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.245703 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:35.244400 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 72903.6us
I20240506 15:14:35.247448 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.219077 20883 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.262861 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.193068 22074 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.251618 22069 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.239953 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.210556 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.240903 22086 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.262602 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.235738 22105 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.262939 20880 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.241792 22097 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:35.262355 20878 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.269498 20876 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.270951 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.250206 20875 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.277523 22019 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.278012 22019 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.278364 22019 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.250725 22046 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.282968 20921 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.283329 20877 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240506 15:14:35.264369 20996 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.274765 20765 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:14:35.264760 22059 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.289999 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 108663us
I20240506 15:14:35.267339 22071 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240506 15:14:35.262226 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49599.1us
I20240506 15:14:35.290426 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.275528 22105 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.273906 22086 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:35.291847 20765 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 55907.1us
I20240506 15:14:35.292405 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.271690 22007 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.270915 22055 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.268916 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.297641 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.275772 22079 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.271886 22155 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.299829 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.302340 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.304637 20874 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.305549 20996 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.307801 22158 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.308480 22158 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.308506 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.308965 22158 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.309967 22084 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.311020 21025 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.307950 20875 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:35.312819 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.299765 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:35.325196 22046 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.311951 21971 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.301301 22007 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.326167 22007 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.327339 22020 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.328981 22021 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.329421 22020 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.329320 21970 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.330958 22021 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.331386 21970 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.303107 22055 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:35.325066 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.295542 22071 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.333123 21027 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.337487 20877 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.337975 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.311987 22084 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.343899 20882 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:35.351902 22146 log.cc:924] Time spent T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20240506 15:14:35.354514 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.355062 20874 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.356887 20996 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.312301 22079 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.325798 22046 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:35.360728 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 77816.1us
I20240506 15:14:35.362548 20921 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.364277 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.365262 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.366039 20882 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:35.369861 22100 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:35.372622 22100 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.377561 22004 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.381197 22103 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.383833 22103 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.340533 22071 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.357781 22160 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240506 15:14:35.312791 22155 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.356406 22020 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.391846 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:35.400494 20845 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.401737 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.402755 20845 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.402909 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.403527 20845 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.404440 22114 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.404681 22129 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.405336 22129 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.405800 22129 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.406307 22119 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.406728 22119 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:35.407099 22119 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.408928 22004 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.408828 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.410060 20875 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:35.414023 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 51541.6us
I20240506 15:14:35.414367 21970 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.414757 21970 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.414767 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.415262 21970 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.415835 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:35.415925 22106 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.416564 21971 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:35.417196 22164 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.417150 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.404848 22114 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.418097 21971 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.418197 20921 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.418455 22114 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.418725 21971 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.419149 21971 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.419406 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.354244 22157 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240506 15:14:35.420130 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55921.7us
I20240506 15:14:35.421375 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.422660 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.364571 22046 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.361840 22079 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.427944 20921 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.428881 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.429638 20845 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.417791 22164 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.403934 20882 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:35.405920 22135 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240506 15:14:35.425782 20924 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.167s (SENT) [suppressed 25 similar messages]
I20240506 15:14:35.425465 20879 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240506 15:14:35.418045 22106 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.420472 22020 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:35.493680 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 67114.2us
I20240506 15:14:35.497023 20921 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.485044 22084 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.488441 22164 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.498566 22167 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.498996 22167 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.499373 22167 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.427958 22096 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:35.430783 22021 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.430497 22131 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.427835 22004 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.396764 22155 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.502336 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.426065 22157 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:35.493695 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 212148us
I20240506 15:14:35.427835 22074 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.396143 22159 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.506877 20921 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.508873 20998 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.487532 20779 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:35.492761 22135 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.502805 22020 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.513481 22074 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.498468 22084 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.428004 22007 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.520664 22074 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.526916 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.513505 22157 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.521214 22084 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.509980 22021 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.507766 22106 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:35.534763 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 105155us
I20240506 15:14:35.508836 22131 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.536221 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.517773 22135 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.509259 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:35.508085 22004 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.531481 21690 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:35.530203 21960 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.513748 22159 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.507197 22103 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:35.541536 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 112711us
I20240506 15:14:35.517066 22096 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:35.549543 21960 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.544761 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.551167 22007 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.539368 22106 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.553659 20996 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:35.549821 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.539148 22021 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.552299 20875 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.551592 22169 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.540045 22131 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.549881 22170 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.560629 21960 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.567685 22166 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20240506 15:14:35.552702 22159 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.570281 22176 consensus_queue.cc:1046] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:35.594098 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.594985 20877 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:14:35.595602 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 88774.4us
I20240506 15:14:35.576884 22068 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.578437 22119 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.544890 22004 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.552935 22103 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.538300 22019 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.576504 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.636392 20874 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.593638 21970 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:35.631870 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 90159.3us
I20240506 15:14:35.637609 20845 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.597740 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.580410 22106 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.640481 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.580050 22163 raft_consensus.cc:2781] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.589910 22169 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.639309 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.644582 22177 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:35.650635 22177 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:35.650573 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.651211 22177 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.651567 20875 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:35.652423 22176 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.652810 22176 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.653127 22176 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.591504 22170 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.648056 20880 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.656679 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 103123us
W20240506 15:14:35.648643 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 50990.6us
I20240506 15:14:35.569629 22152 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.654366 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:35.654114 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 125185us
I20240506 15:14:35.661360 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.658742 22074 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:35.645876 22160 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20240506 15:14:35.638587 22103 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.675331 20921 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:35.680701 20904 debug-util.cc:398] Leaking SignalData structure 0x7b08001d6c20 after lost signal to thread 19729
I20240506 15:14:35.681231 21970 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.681294 20952 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20240506 15:14:35.681784 20904 debug-util.cc:398] Leaking SignalData structure 0x7b08001986c0 after lost signal to thread 20814
W20240506 15:14:35.683166 20904 debug-util.cc:398] Leaking SignalData structure 0x7b0800229480 after lost signal to thread 20907
I20240506 15:14:35.648093 22163 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:35.638733 22171 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.691135 22100 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:35.654600 22169 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.659879 22170 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.724784 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 87225.9us
I20240506 15:14:35.725723 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.716086 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.730401 20878 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:35.700950 20904 debug-util.cc:398] Leaking SignalData structure 0x7b08002fd4e0 after lost signal to thread 20982
W20240506 15:14:35.743352 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80385.2us
I20240506 15:14:35.747313 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:35.765395 20881 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.677898 22173 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:35.704761 22163 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:35.788944 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.789863 20958 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.789026 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:35.791262 20957 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.794502 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.709139 22171 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:35.719635 22096 raft_consensus.cc:686] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:35.784008 22148 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240506 15:14:35.803637 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.804493 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.805174 20952 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.810055 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.812208 20950 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.785686 22173 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:35.816761 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 91129.6us
I20240506 15:14:35.780643 22158 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.800535 22171 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.876070 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.876991 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.883476 20953 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:35.879426 22020 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.880370 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:35.819764 20904 debug-util.cc:398] Leaking SignalData structure 0x7b0800133100 after lost signal to thread 21056
I20240506 15:14:35.884565 20950 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.882503 20880 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:35.888064 20956 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.891556 21684 raft_consensus.cc:1226] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:35.891820 22185 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.905078 20874 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:35.905962 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.906307 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.817907 22173 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:35.911206 20953 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.911043 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.911839 20950 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:35.940099 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.942220 20956 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.889448 22068 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:35.917312 22165 log.cc:924] Time spent T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.378s user 0.002s sys 0.000s
I20240506 15:14:35.894209 22185 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:35.945995 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 270731us
I20240506 15:14:35.947338 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.931953 20881 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
W20240506 15:14:35.950517 21971 consensus_meta.cc:287] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.523s user 0.041s sys 0.034s
W20240506 15:14:35.950757 22105 consensus_meta.cc:287] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.647s user 0.015s sys 0.000s
W20240506 15:14:35.951617 22071 consensus_meta.cc:287] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.531s user 0.017s sys 0.049s
I20240506 15:14:35.956573 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.961549 22158 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.893797 22119 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.963831 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.964869 20956 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:35.974018 20921 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.904899 22019 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:35.952574 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:35.973165 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 155689us
I20240506 15:14:35.974918 20921 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:35.972141 22071 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.969748 21971 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.963096 22152 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:35.912851 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.943989 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.953711 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:35.995020 20845 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:35.971357 20904 debug-util.cc:398] Leaking SignalData structure 0x7b0800322820 after lost signal to thread 21718
I20240506 15:14:35.962983 22096 consensus_queue.cc:237] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.006006 20845 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.987016 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.026605 20956 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:36.027737 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:35.962746 22183 raft_consensus.cc:2741] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240506 15:14:36.023594 20952 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.023881 22188 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.024016 20955 raft_consensus.cc:2354] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:35.965142 22185 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:35.984143 22068 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:35.945490 22084 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:35.969774 22105 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.033887 20845 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:35.983863 22074 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.035053 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.036674 20950 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:36.027261 20904 debug-util.cc:398] Leaking SignalData structure 0x7b08003226e0 after lost signal to thread 21719
I20240506 15:14:36.022579 22019 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.039762 22188 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.038024 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:36.061902 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.061934 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:36.047962 22086 consensus_meta.cc:287] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.716s user 0.006s sys 0.017s
I20240506 15:14:36.070771 20957 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:36.082046 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.072139 20878 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
W20240506 15:14:36.049582 22161 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.014s sys 0.039s
I20240506 15:14:36.050978 22068 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:36.064834 20904 debug-util.cc:398] Leaking SignalData structure 0x7b0800322580 after lost signal to thread 21972
I20240506 15:14:36.033756 21970 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240506 15:14:36.058311 22172 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:36.086992 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:36.087612 22161 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s user 0.015s sys 0.039s
I20240506 15:14:36.085990 22020 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.091097 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.092432 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:36.092804 22114 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.674s user 0.013s sys 0.006s
I20240506 15:14:36.061784 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:36.063416 22019 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:36.084844 22135 consensus_meta.cc:287] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.529s user 0.023s sys 0.036s
I20240506 15:14:36.075212 22188 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:36.094858 20904 debug-util.cc:398] Leaking SignalData structure 0x7b0800322860 after lost signal to thread 22009
I20240506 15:14:36.101364 20846 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.098699 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.129998 20950 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:36.093178 22186 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:36.095616 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.131007 20949 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
W20240506 15:14:36.094094 22055 consensus_meta.cc:287] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.758s user 0.012s sys 0.004s
I20240506 15:14:36.089522 22068 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.123862 22193 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.070266 22007 consensus_queue.cc:1046] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20240506 15:14:36.143803 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.096879 21971 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.146503 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.146993 20955 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.093458 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:36.148186 22069 consensus_meta.cc:287] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.867s user 0.011s sys 0.000s
I20240506 15:14:36.149245 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:36.151052 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 349448us
W20240506 15:14:36.154455 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 127786us
I20240506 15:14:36.130239 22019 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.099584 22172 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:36.086009 22086 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.080453 22159 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.146522 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.093565 22192 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:36.134485 22186 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:36.169327 20998 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:36.155925 20904 debug-util.cc:398] Leaking SignalData structure 0x7b08000c09e0 after lost signal to thread 22061
I20240506 15:14:36.172528 20949 raft_consensus.cc:2379] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.136406 22135 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.171871 21971 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20240506 15:14:36.132393 22114 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.188848 20998 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.150210 22055 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.180099 22172 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:36.173880 22084 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:36.182714 22059 consensus_meta.cc:287] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.870s user 0.003s sys 0.009s
I20240506 15:14:36.194470 20953 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.204378 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 66) took 1821 ms. Trace:
W20240506 15:14:36.213966 22079 consensus_meta.cc:287] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.707s user 0.014s sys 0.003s
I20240506 15:14:36.207993 22059 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.214541 22079 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.186605 20952 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:36.216130 22059 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.197556 22119 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.216475 20882 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.207665 20780 rpcz_store.cc:276] 0506 15:14:34.374700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:34.386821 (+ 12121us) service_pool.cc:225] Handling call
0506 15:14:36.195797 (+1808976us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:35.200928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:35.208815 (+ 7887us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:35.208877 (+ 62us) write_op.cc:435] Acquired schema lock
0506 15:14:35.208898 (+ 21us) tablet.cc:660] Decoding operations
0506 15:14:35.209316 (+ 418us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:35.209446 (+ 130us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:35.209479 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:35.209801 (+ 322us) tablet.cc:699] Row locks acquired
0506 15:14:35.209820 (+ 19us) write_op.cc:260] PREPARE: finished
0506 15:14:35.209989 (+ 169us) write_op.cc:270] Start()
0506 15:14:35.210128 (+ 139us) write_op.cc:276] Timestamp: P: 1715008475209923 usec, L: 0
0506 15:14:35.210159 (+ 31us) op_driver.cc:331] REPLICATION: starting
0506 15:14:35.210889 (+ 730us) log.cc:842] Serialized 535 byte log entry
0506 15:14:35.588353 (+377464us) op_driver.cc:444] REPLICATION: finished
0506 15:14:35.915816 (+327463us) write_op.cc:301] APPLY: starting
0506 15:14:35.938469 (+ 22653us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:35.948212 (+ 9743us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:35.996502 (+ 48290us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:36.047643 (+ 51141us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:36.063885 (+ 16242us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:36.074729 (+ 10844us) write_op.cc:312] APPLY: finished
0506 15:14:36.147004 (+ 72275us) log.cc:842] Serialized 22 byte log entry
0506 15:14:36.147188 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:36.147263 (+ 75us) write_op.cc:454] Released schema lock
0506 15:14:36.147301 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:36.147621 (+ 320us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:36.147646 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":229715,"apply.run_cpu_time_us":3005,"apply.run_wall_time_us":258568,"num_ops":1,"prepare.queue_time_us":244230,"prepare.run_cpu_time_us":15773,"prepare.run_wall_time_us":70394,"raft.queue_time_us":35979,"raft.run_cpu_time_us":94027,"raft.run_wall_time_us":381231,"replication_time_us":378102,"thread_start_us":444253,"threads_started":4,"wal-append.queue_time_us":78240}]]}
W20240506 15:14:36.219288 22046 consensus_meta.cc:287] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.713s user 0.009s sys 0.006s
I20240506 15:14:36.219516 22055 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.205191 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.193571 22007 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.220171 22007 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:36.220577 22007 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.185480 22192 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.222429 22192 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.224006 21690 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:36.185338 22186 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.218055 22119 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:36.226897 22021 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.636s user 0.008s sys 0.001s
I20240506 15:14:36.227303 22021 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.229341 20954 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.172325 22069 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.233269 22021 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.201282 22198 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:36.229779 22046 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:36.224318 22176 consensus_meta.cc:287] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.003s
I20240506 15:14:36.184784 22187 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240506 15:14:36.244185 22079 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.205704 22196 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.225250 22071 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.231751 22119 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.261245 20954 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.239519 22105 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:36.278139 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 253198us
I20240506 15:14:36.258922 22161 raft_consensus.cc:3032] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
W20240506 15:14:36.284161 22167 consensus_meta.cc:287] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.784s user 0.012s sys 0.009s
I20240506 15:14:36.284605 22167 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.285115 22046 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.271450 22021 consensus_queue.cc:1046] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20240506 15:14:36.286542 22167 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.286617 22021 raft_consensus.cc:3032] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:36.287184 22021 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.288509 20955 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
W20240506 15:14:36.289086 22195 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20240506 15:14:36.289911 22069 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:36.282797 22103 consensus_meta.cc:287] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.512s user 0.007s sys 0.002s
W20240506 15:14:36.277570 22129 consensus_meta.cc:287] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.871s user 0.013s sys 0.000s
W20240506 15:14:36.297525 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 218446us
I20240506 15:14:36.305392 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:36.288630 22004 consensus_meta.cc:287] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.633s user 0.009s sys 0.000s
I20240506 15:14:36.272512 22196 raft_consensus.cc:686] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:36.252288 22176 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.283231 22161 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.269335 22019 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:36.309373 22164 consensus_meta.cc:287] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.811s user 0.012s sys 0.000s
I20240506 15:14:36.376112 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:36.377262 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 233513us
I20240506 15:14:36.288172 22021 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.378999 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.379681 22100 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:36.379756 22021 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:36.380012 22191 raft_consensus.cc:3032] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:36.380448 22191 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:36.380234 22100 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.381274 22191 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.382345 22100 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.382504 22191 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:36.386601 22046 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.386687 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:36.387149 22046 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.387517 22046 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.387472 22183 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.388185 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 192) took 1062 ms. Trace:
I20240506 15:14:36.388628 20779 rpcz_store.cc:276] 0506 15:14:35.326003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:35.337469 (+ 11466us) service_pool.cc:225] Handling call
0506 15:14:36.388114 (+1050645us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:35.542652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:35.577031 (+ 34379us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:35.577082 (+ 51us) write_op.cc:435] Acquired schema lock
0506 15:14:35.577109 (+ 27us) tablet.cc:660] Decoding operations
0506 15:14:35.577546 (+ 437us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:35.577618 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:35.577659 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:35.577864 (+ 205us) tablet.cc:699] Row locks acquired
0506 15:14:35.577890 (+ 26us) write_op.cc:260] PREPARE: finished
0506 15:14:35.578073 (+ 183us) write_op.cc:270] Start()
0506 15:14:35.578214 (+ 141us) write_op.cc:276] Timestamp: P: 1715008475578011 usec, L: 0
0506 15:14:35.578246 (+ 32us) op_driver.cc:331] REPLICATION: starting
0506 15:14:35.579021 (+ 775us) log.cc:842] Serialized 535 byte log entry
0506 15:14:35.811009 (+231988us) op_driver.cc:444] REPLICATION: finished
0506 15:14:36.084602 (+273593us) write_op.cc:301] APPLY: starting
0506 15:14:36.088988 (+ 4386us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:36.124292 (+ 35304us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:36.161090 (+ 36798us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:36.201185 (+ 40095us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:36.236145 (+ 34960us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:36.241752 (+ 5607us) write_op.cc:312] APPLY: finished
0506 15:14:36.277321 (+ 35569us) log.cc:842] Serialized 22 byte log entry
0506 15:14:36.305087 (+ 27766us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:36.310227 (+ 5140us) write_op.cc:454] Released schema lock
0506 15:14:36.376929 (+ 66702us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:36.379199 (+ 2270us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:36.379232 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":244098,"apply.run_cpu_time_us":17417,"apply.run_wall_time_us":304773,"mutex_wait_us":2628,"num_ops":1,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":3122,"prepare.run_wall_time_us":43866,"replication_time_us":224127,"thread_start_us":99109,"threads_started":1}]]}
I20240506 15:14:36.393388 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.384682 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:36.395423 22071 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:36.395836 22071 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:36.396206 22071 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.377090 22069 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.396750 22069 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.397104 22069 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.309689 22103 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.402114 20950 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.312180 22129 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.404728 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:36.394862 22131 consensus_meta.cc:287] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.804s user 0.009s sys 0.004s
W20240506 15:14:36.408386 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 255814us
I20240506 15:14:36.327692 22004 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.411792 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:36.400074 22157 consensus_meta.cc:287] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.850s user 0.008s sys 0.004s
I20240506 15:14:36.413055 20996 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:36.415143 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.415416 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.435654 20880 raft_consensus.cc:2362] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
W20240506 15:14:36.416150 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 247711us
I20240506 15:14:36.416550 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.416850 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.417151 20951 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.417569 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:36.418444 22046 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.419242 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.419718 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.421672 22004 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.445356 21032 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.424424 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.426152 22176 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:36.406826 21960 consensus_meta.cc:287] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.767s user 0.008s sys 0.013s
I20240506 15:14:36.446992 20883 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:36.447227 21960 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.447942 22068 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.428789 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.429328 20953 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.419874 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:36.429469 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.019s sys 0.022s
I20240506 15:14:36.404188 22193 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:36.437558 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.350520 22164 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.419854 22007 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.439922 21027 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240506 15:14:36.442092 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.404978 22183 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:36.443961 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.461298 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:36.441896 22106 consensus_meta.cc:287] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.772s user 0.016s sys 0.000s
I20240506 15:14:36.446085 22185 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:36.454262 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.020s sys 0.023s
I20240506 15:14:36.448616 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.472409 20876 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.472491 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.414050 22086 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:36.478433 22169 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.708s user 0.010s sys 0.000s
I20240506 15:14:36.478814 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.479449 21029 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.479161 22169 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.462821 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.457084 21023 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:36.465855 22164 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.466472 20958 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.433259 22159 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.493674 22193 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.448524 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.448122 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.436053 22096 raft_consensus.cc:2741] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240506 15:14:36.438177 22114 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.468945 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:36.449156 22163 consensus_meta.cc:287] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.654s user 0.008s sys 0.004s
I20240506 15:14:36.470564 22198 raft_consensus.cc:2741] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20240506 15:14:36.445178 22170 consensus_meta.cc:287] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.665s user 0.004s sys 0.007s
I20240506 15:14:36.457468 20882 raft_consensus.cc:2362] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4. Re-sending same reply.
I20240506 15:14:36.443743 21025 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.485432 20957 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.512820 21028 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.513306 22007 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.514335 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.514943 20874 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:36.485138 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.488358 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.517131 21029 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.449388 22161 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.415673 22131 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.478539 22019 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.443123 20924 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.449278 21960 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.499876 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.502152 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.525449 20845 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.525765 21031 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.502550 20876 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.450413 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.503057 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.440220 22135 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.436748 22157 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.506209 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.507051 20881 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.467325 22183 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:36.503508 22155 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.981s user 0.008s sys 0.011s
I20240506 15:14:36.518347 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.491758 22172 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.513940 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.521790 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.536368 21029 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.449003 22196 consensus_queue.cc:237] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:36.523663 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80603.2us
I20240506 15:14:36.527655 20845 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.483110 22106 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.528887 21027 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.528281 21970 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:36.515578 22046 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.530041 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.534801 20953 raft_consensus.cc:2354] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.550470 20845 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.522961 21032 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:36.539870 21028 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.541043 20949 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.528472 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.542414 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.526504 22198 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:36.531637 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.546089 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.508296 22163 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.549012 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:36.545152 22204 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:36.534703 22183 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:36.557294 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.569972 21032 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.570106 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.515477 22170 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.558455 20958 raft_consensus.cc:2379] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.555459 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.566069 20874 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.571633 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.561874 20776 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 2, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 2 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:36.572207 20953 raft_consensus.cc:2354] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.567829 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:36.536284 22161 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.520447 22069 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.573518 20950 raft_consensus.cc:2354] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.573027 20776 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 4 to 5, term changed from 3 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:36.564692 20955 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.568907 22198 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.543301 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.545471 22135 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.582636 20952 raft_consensus.cc:1226] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.576756 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.581696 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.583786 21032 raft_consensus.cc:2354] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:36.577693 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.584880 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.585012 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.569320 22103 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.577538 21685 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:36.575426 21970 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.573340 21960 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:36.589145 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.608631 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.609131 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.575207 22114 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.592978 21029 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.594591 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.580026 22129 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.575618 22194 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:36.578089 22164 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:36.601855 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.542441 22155 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.609273 20950 raft_consensus.cc:2354] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:36.579912 22198 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:36.612777 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.639475 20952 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.615401 22200 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:36.615814 20879 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.645442 22200 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.616490 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.600704 22068 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.617166 20958 raft_consensus.cc:2354] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.615782 21031 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.617974 22131 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.582659 22135 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.588234 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.653679 20953 raft_consensus.cc:2354] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
W20240506 15:14:36.621645 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 53861us
I20240506 15:14:36.621650 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.624794 21032 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:36.645148 22171 consensus_meta.cc:287] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.755s user 0.009s sys 0.002s
I20240506 15:14:36.654764 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:36.625185 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.626025 22103 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:36.627087 21960 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.656533 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:36.628640 22185 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.628899 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.633724 22069 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.634017 22186 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.621008 22114 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.616465 22169 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.635330 22187 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:36.670347 22187 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.670744 22187 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:36.671062 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.671794 20880 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.615427 22163 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.643874 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.672639 20950 raft_consensus.cc:2354] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.615945 20951 raft_consensus.cc:2354] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.586745 22201 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.647279 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.602603 22004 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:36.650135 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.657990 22135 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.677103 20952 raft_consensus.cc:2379] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:36.624171 22177 consensus_meta.cc:287] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.972s user 0.012s sys 0.006s
I20240506 15:14:36.616443 22170 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.608273 21970 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:36.621850 22172 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.675098 20955 raft_consensus.cc:2354] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.656291 20949 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.657109 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.685286 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.659950 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.638635 22194 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:36.645768 22203 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:36.675756 22114 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.689584 21028 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.684800 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.690670 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.691061 20882 raft_consensus.cc:2379] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.691228 22157 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.692001 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.693549 20952 raft_consensus.cc:2354] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:36.664538 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.694283 20876 raft_consensus.cc:2354] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.665091 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.658654 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.666915 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.694936 20881 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.695186 20954 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.667196 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.695855 20957 raft_consensus.cc:2354] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.668215 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.668399 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.696683 20953 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.614849 22119 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.672938 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.651261 22200 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.649128 21025 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.707324 22186 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:36.658134 21026 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:36.682078 22177 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.692656 22194 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:36.661435 22131 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.662734 22204 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.682399 21024 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:36.654764 21029 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.686269 21031 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.717788 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.718168 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.718557 21031 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.718971 20953 raft_consensus.cc:2379] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.683899 21970 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.688103 21960 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:36.695282 20883 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.679562 22201 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:36.700906 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.701578 20874 raft_consensus.cc:2354] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.702055 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.722774 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.723553 20883 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.663483 22171 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.704190 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.704109 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.705263 22192 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.705583 22106 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.706219 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:36.707269 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.674746 22185 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:36.709563 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.678344 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.711479 20881 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
W20240506 15:14:36.736681 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 58356.8us
I20240506 15:14:36.712980 21027 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.686196 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.679370 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.717790 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.663983 22071 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.720705 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.739900 20880 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.740993 21029 raft_consensus.cc:2379] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.741091 20949 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:36.701375 22163 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.724493 20950 raft_consensus.cc:2379] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.679726 22004 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:36.726729 20952 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.710777 21028 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.759194 22114 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.735702 22155 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.761097 20877 raft_consensus.cc:2354] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.726816 21032 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.723277 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.729084 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.730664 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.764008 20876 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.712548 21023 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.733250 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.716881 22208 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:36.736996 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:36.769117 20953 raft_consensus.cc:2354] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.736800 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.769793 20957 raft_consensus.cc:2354] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.770078 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.770502 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.771215 20957 raft_consensus.cc:2379] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.771070 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.771968 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.772207 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.772759 20957 raft_consensus.cc:2379] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.773032 20950 raft_consensus.cc:2379] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.773473 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.773842 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.774641 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.775328 20957 raft_consensus.cc:2354] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
W20240506 15:14:36.778944 22173 consensus_meta.cc:287] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.838s user 0.010s sys 0.002s
W20240506 15:14:36.779654 22205 log.cc:924] Time spent T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.299s user 0.006s sys 0.000s
I20240506 15:14:36.779590 22173 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.780881 20956 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.781275 20879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1138) took 1251 ms. Trace:
I20240506 15:14:36.781580 22177 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:36.739627 22119 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.726164 22201 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.792770 22172 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:36.794381 20921 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.739205 22103 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:36.745821 20875 raft_consensus.cc:2379] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.745852 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.746913 20845 leader_election.cc:398] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:36.800448 20845 leader_election.cc:401] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.753576 20998 leader_election.cc:398] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.754222 22004 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:36.755575 22188 consensus_meta.cc:287] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.003s
I20240506 15:14:36.756446 22163 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.757511 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:36.765436 20874 raft_consensus.cc:2379] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.766597 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.802489 21027 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.766542 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.802798 20880 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.768317 22208 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.681757 22207 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240506 15:14:36.784456 20996 leader_election.cc:398] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.811887 22069 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.737517 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.813724 22207 consensus_queue.cc:1225] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240506 15:14:36.776831 21026 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:36.814386 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.815141 20874 raft_consensus.cc:2354] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:36.815477 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.737823 21025 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.812353 22103 raft_consensus.cc:686] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.817210 22173 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.816864 22163 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.817796 22103 consensus_queue.cc:237] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.752853 22185 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.803938 22188 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.819958 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.820464 22188 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.821117 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.821892 21026 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.822291 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.822851 20880 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.815865 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.824923 20874 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:36.796629 20921 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.797888 20883 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:36.800932 20998 leader_election.cc:401] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240506 15:14:36.826524 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 81316.7us
I20240506 15:14:36.826731 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.827098 20921 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.827510 20883 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.827428 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:36.801692 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 58311.5us
I20240506 15:14:36.762903 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:36.828362 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.828655 20952 raft_consensus.cc:2354] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.796553 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:36.805801 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.828925 20883 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:36.814304 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 76855us
I20240506 15:14:36.797529 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.790496 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.829674 21024 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.829869 21029 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.813843 20996 leader_election.cc:401] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.781695 20879 rpcz_store.cc:276] 0506 15:14:35.529508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:35.530854 (+ 1346us) service_pool.cc:225] Handling call
0506 15:14:36.781061 (+1250207us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400181354. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:36.781196 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2335910272}
I20240506 15:14:36.830984 22068 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.831523 22068 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.831655 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.832630 20879 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.832438 22068 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.833971 22164 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.834903 22177 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:36.835583 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 51378.4us
I20240506 15:14:36.789839 22071 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:36.837164 20996 leader_election.cc:398] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.837299 20878 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.837507 20996 leader_election.cc:401] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.806663 21032 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.821162 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:36.823963 21030 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:36.828027 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:36.837522 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.839205 20877 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.834614 22198 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.816023 22114 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.840706 20998 leader_election.cc:398] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.843298 20998 leader_election.cc:401] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.843645 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.844053 20996 leader_election.cc:398] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.843961 20998 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.845319 21027 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.850124 20876 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:36.836848 22183 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.859508 22183 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.807193 22119 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:36.849903 22155 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.860198 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.865262 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.865995 21031 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.800518 22192 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.866969 20876 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:36.862804 21029 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:36.852447 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.868440 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.868590 21024 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:36.869719 22019 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:36.869956 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.870222 22019 raft_consensus.cc:686] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.863464 22185 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.862866 22187 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.872166 20954 raft_consensus.cc:2354] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:36.872997 21028 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.873960 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.870606 21031 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:36.875068 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47131.9us
I20240506 15:14:36.887519 20921 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.868360 20883 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:36.874784 22071 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.838603 22161 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:36.817808 22173 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.860860 22208 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.814394 22207 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:36.904959 22207 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:36.897800 22135 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.905894 22207 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:36.907182 22207 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:36.882162 20996 leader_election.cc:398] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:36.907634 22135 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.907974 20996 leader_election.cc:401] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.908834 20996 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.909963 20996 leader_election.cc:398] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.910210 20996 leader_election.cc:401] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.865846 22191 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.888233 20921 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.923842 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.889139 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.892688 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.891453 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.925745 20921 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.871155 22019 consensus_queue.cc:237] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.877293 20877 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
W20240506 15:14:36.876158 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 106919us
I20240506 15:14:36.895577 20845 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:36.899365 21027 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:36.928283 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.929256 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.901572 20874 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:36.898284 22046 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.880107 20875 raft_consensus.cc:2354] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:36.880618 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.902748 22071 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.951917 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.903395 22187 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.903437 20883 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:36.900161 22161 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.904462 22173 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:36.956856 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:36.955945 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.912103 20881 raft_consensus.cc:2354] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.911165 20996 leader_election.cc:398] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
W20240506 15:14:36.960788 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 49732.2us
I20240506 15:14:36.961920 20996 leader_election.cc:398] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.962167 20996 leader_election.cc:401] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.877336 22201 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.913019 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.902194 22194 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.913610 22157 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.913997 22200 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.878719 22198 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.899792 22193 consensus_queue.cc:1046] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240506 15:14:36.879058 22169 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.916040 21025 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:36.917703 22186 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:36.918447 21023 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.938721 21031 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:36.974964 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:36.940860 20998 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.942023 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.933730 21653 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:36.944074 21030 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:36.943238 20880 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:36.964470 22171 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.953445 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
W20240506 15:14:36.983047 22211 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20240506 15:14:36.984071 22196 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.984947 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.985510 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.986075 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.986819 21031 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:36.987053 21025 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:36.986955 20780 catalog_manager.cc:5484] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:36.987694 21031 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:36.940171 22208 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.883419 20924 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:36.990505 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 107124us
I20240506 15:14:36.979590 20845 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:36.991843 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.932833 22135 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:36.939951 22192 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.944787 22164 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.966352 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:36.965608 22176 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20240506 15:14:37.000167 20996 leader_election.cc:398] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.000531 20996 leader_election.cc:401] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.001480 20996 leader_election.cc:398] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.001562 22201 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:36.950805 22021 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.968675 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:36.972028 20950 raft_consensus.cc:2354] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.973439 20953 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.962889 21026 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:36.977370 20998 leader_election.cc:398] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:36.894192 22177 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:36.976605 22169 raft_consensus.cc:686] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:36.967911 22167 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.968053 22086 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:36.966353 22007 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.016609 21029 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:36.980221 20882 raft_consensus.cc:2354] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:36.981877 22173 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:36.982708 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:36.993410 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:36.980245 22055 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:36.972725 22106 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.001780 20996 leader_election.cc:401] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.003105 22171 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.004037 21024 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:37.005101 22157 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.008244 20998 leader_election.cc:401] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240506 15:14:36.978520 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.002174 22196 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.005218 22208 raft_consensus.cc:686] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.022928 20952 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:36.944945 22069 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:36.994715 22103 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.007340 22068 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.032177 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:37.025190 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:37.032974 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55729.3us
W20240506 15:14:37.095520 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 63397.3us
I20240506 15:14:37.033895 20996 leader_election.cc:398] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:37.005254 22164 raft_consensus.cc:686] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.010763 22192 raft_consensus.cc:686] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.033042 21027 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.095367 22193 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:37.098382 20846 leader_election.cc:398] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 2. Current term is 3.
I20240506 15:14:37.104557 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.105084 20998 leader_election.cc:398] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.106300 20998 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.096594 20996 leader_election.cc:401] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.025419 22021 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.027849 22007 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.020272 22055 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:37.029726 22155 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:37.026782 22172 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:37.056418 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.096531 22069 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.114711 21028 raft_consensus.cc:2379] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:37.058176 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.044888 22167 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:37.118495 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 92736.2us
I20240506 15:14:37.101855 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:37.099697 22187 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.122370 20954 raft_consensus.cc:2354] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.107841 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.046046 22171 raft_consensus.cc:686] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.125413 20952 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.135859 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.121006 22055 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:37.028067 22071 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.100679 22086 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.104054 22186 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:37.127936 20996 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.131067 20953 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:37.142872 20922 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:37.146569 20996 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:14:37.147835 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 153622us
W20240506 15:14:37.102021 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 118380us
I20240506 15:14:37.115311 22157 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.110772 22193 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:37.153498 22193 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.125712 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.146417 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.123978 22185 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.121698 22196 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:37.139838 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.191236 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.139401 22194 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:37.194001 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:37.156558 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:37.156781 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:37.121876 22172 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.144870 22159 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.144004 22200 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:37.146919 22007 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.198472 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.121479 22021 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.144798 22187 raft_consensus.cc:686] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.155766 22173 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:37.198458 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 51934.2us
I20240506 15:14:37.205312 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.206079 20996 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.189055 22183 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.189859 22103 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:37.207376 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.192108 21028 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:37.210052 20996 leader_election.cc:398] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.210391 20996 leader_election.cc:401] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.136806 22169 consensus_queue.cc:237] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.223517 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:37.239408 21024 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.218132 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:37.248525 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.203547 22196 raft_consensus.cc:2741] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240506 15:14:37.204506 22185 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:37.146759 22208 consensus_queue.cc:237] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.254119 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.196468 21656 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.207847 22222 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.214723 22119 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.236323 22200 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.275817 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.276726 20953 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.279058 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.280174 20954 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.196796 22164 consensus_queue.cc:237] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.254807 22185 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:37.264220 22159 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:37.264696 22183 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:37.266036 22046 raft_consensus.cc:686] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.295563 22226 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:37.296526 22226 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.299758 22226 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:37.304529 22226 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.305392 22226 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:37.306913 22226 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.218082 22103 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.271719 21960 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:37.276590 22007 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.327669 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.328923 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.329644 20956 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.310647 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.330377 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.331092 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.331920 20953 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:37.285228 21970 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.558s user 0.011s sys 0.000s
I20240506 15:14:37.332746 22159 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.333488 20949 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.334764 20949 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:37.334929 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:37.214443 22192 consensus_queue.cc:237] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.335875 20952 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:37.326731 22103 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:37.315871 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.338936 20954 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:37.268124 22188 consensus_queue.cc:1046] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240506 15:14:37.291846 22185 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.332998 21970 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.229748 22086 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.291517 22183 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:37.331421 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.215788 22069 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.230011 22167 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.364018 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.365293 20955 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:37.332757 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.366981 20953 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:37.285107 22227 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.285357 22119 consensus_queue.cc:1046] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20240506 15:14:37.236074 22171 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.265595 22187 consensus_queue.cc:237] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.358695 21970 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:37.369151 22183 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:37.373819 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 174250us
I20240506 15:14:37.286583 22172 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.246511 22161 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:37.376856 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 106879us
W20240506 15:14:37.382630 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 191783us
W20240506 15:14:37.387482 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 177532us
I20240506 15:14:37.389289 20953 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:37.388129 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.390262 20996 leader_election.cc:398] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.391376 21656 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:37.392918 20998 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:37.387475 22004 consensus_meta.cc:287] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.566s user 0.003s sys 0.008s
I20240506 15:14:37.395324 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.386427 22046 consensus_queue.cc:237] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.398610 22198 raft_consensus.cc:3032] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:37.399039 22198 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.399241 22007 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.400381 22191 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.401952 22055 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.404374 22220 consensus_queue.cc:1046] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:37.407644 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.390888 20996 leader_election.cc:401] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.408175 21024 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:37.389890 22161 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:37.390537 22200 consensus_queue.cc:1046] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240506 15:14:37.412343 22004 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.422741 20998 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.450431 20998 leader_election.cc:398] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:37.451283 20998 leader_election.cc:398] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:37.451689 20998 leader_election.cc:401] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.397140 22227 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.412045 22119 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.440793 21029 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240506 15:14:37.411505 22207 consensus_meta.cc:287] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.504s user 0.003s sys 0.009s
I20240506 15:14:37.421177 20996 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.409879 21026 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:37.411656 22055 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:37.438433 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1203) took 1015 ms. Trace:
I20240506 15:14:37.410531 22191 raft_consensus.cc:686] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:37.454831 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.410629 22221 raft_consensus.cc:3032] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:37.455924 20846 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.460108 21024 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:37.462826 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:37.471215 20881 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:37.472895 21653 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:37.421624 22231 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:37.474546 20878 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240506 15:14:37.478055 20878 pending_rounds.cc:76] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 5
I20240506 15:14:37.478437 20878 pending_rounds.cc:97] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240506 15:14:37.478682 20878 raft_consensus.cc:2866] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:37.445132 21025 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.476052 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.445312 22220 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.481047 20998 leader_election.cc:398] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.483911 20998 leader_election.cc:401] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.482503 20846 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.449790 20955 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:37.484500 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.485323 20958 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:37.486027 22228 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20240506 15:14:37.485903 22055 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.487788 22191 consensus_queue.cc:237] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.411034 22193 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.468783 22196 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.468744 22007 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.474035 22221 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:37.419667 22230 consensus_queue.cc:1046] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20240506 15:14:37.481839 22119 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:37.475159 22233 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20240506 15:14:37.519167 22233 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.453096 22114 consensus_queue.cc:1046] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240506 15:14:37.509685 22159 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.524140 20958 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:37.493472 22207 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.472548 22200 consensus_queue.cc:1046] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20240506 15:14:37.472903 22188 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240506 15:14:37.491644 22231 raft_consensus.cc:686] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:37.501682 22055 raft_consensus.cc:686] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:37.514971 22220 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.508522 22227 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:37.470407 21023 rpcz_store.cc:276] 0506 15:14:36.422770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.488013 (+ 65243us) service_pool.cc:225] Handling call
0506 15:14:37.438202 (+950189us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540026f7d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:37.438356 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":444731136}
I20240506 15:14:37.551012 22232 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:37.563582 22227 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:37.565131 22235 log.cc:924] Time spent T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20240506 15:14:37.565640 22172 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:37.525698 22007 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:37.563761 22232 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.560545 21656 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
W20240506 15:14:37.567972 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 83548.4us
I20240506 15:14:37.576141 20996 leader_election.cc:398] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
W20240506 15:14:37.577935 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 48510.2us
I20240506 15:14:37.553839 22119 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.596871 22221 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.607335 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.608013 22221 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:37.615677 22004 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:37.616990 22229 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20240506 15:14:37.619833 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.597431 22207 raft_consensus.cc:3032] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
W20240506 15:14:37.652717 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 171778us
I20240506 15:14:37.601593 21028 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:37.680757 20881 raft_consensus.cc:1261] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:37.682597 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:37.688921 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.689642 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.690259 21024 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:37.692425 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.608819 22119 consensus_queue.cc:1046] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20240506 15:14:37.693312 21029 raft_consensus.cc:2354] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4.
I20240506 15:14:37.631354 22207 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.695794 21029 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:37.700150 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:37.660682 22233 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:37.705281 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:37.608896 22177 consensus_queue.cc:1159] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ecededcd854842d4867fda9fa63a4b66" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20240506 15:14:37.677935 22155 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:37.706324 20921 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:37.661957 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 58249.4us
I20240506 15:14:37.712385 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:37.709839 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:37.701289 22237 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20240506 15:14:37.712505 22224 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.714689 21030 raft_consensus.cc:2354] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:37.695519 22222 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:37.690282 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.703840 22167 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:37.701135 22218 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.610328 22007 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.709353 22193 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:37.702761 22231 consensus_queue.cc:237] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.748497 22167 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:37.750020 20828 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240506 15:14:37.749603 22167 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.703709 22069 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240506 15:14:37.750407 22246 consensus_queue.cc:1035] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:37.751277 22069 raft_consensus.cc:3032] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:37.751660 22069 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.734088 21028 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:37.721329 22086 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:37.753271 22086 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.752821 22069 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.754056 22086 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.736471 22207 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.755051 21960 raft_consensus.cc:3032] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:37.755466 21960 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.660059 22055 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.756191 21960 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.758793 22208 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:37.737943 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.738379 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:37.738792 21026 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:37.739787 22232 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:37.740896 20953 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:37.710958 21024 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:37.743969 21029 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:37.703831 20875 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.767539 22218 raft_consensus.cc:686] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.751992 21025 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:37.739460 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:37.768383 20951 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:37.759367 22208 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:37.704289 22163 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20240506 15:14:37.762346 20921 leader_election.cc:398] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4.
I20240506 15:14:37.776444 20921 leader_election.cc:401] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20240506 15:14:37.771636 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 35b78f712a194a51abc96c8500a9a611, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:37.777660 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 35b78f712a194a51abc96c8500a9a611, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.754806 22069 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:37.764153 22224 raft_consensus.cc:686] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:37.788623 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 45277.5us
W20240506 15:14:37.789850 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 10845b6e0cc149f5a5e40605a2ea7e49, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.792109 20921 leader_election.cc:398] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
W20240506 15:14:37.792287 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 10845b6e0cc149f5a5e40605a2ea7e49, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.792457 20921 leader_election.cc:401] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.789623 21970 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:37.823807 21027 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:37.789916 22071 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4.
I20240506 15:14:37.831727 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 915) took 1174 ms. Trace:
I20240506 15:14:37.838048 20958 rpcz_store.cc:276] 0506 15:14:36.646462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.652762 (+ 6300us) service_pool.cc:225] Handling call
0506 15:14:37.820440 (+1167678us) spinlock_profiling.cc:243] Waited 788 ms on lock 0x7b540012e654. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:37.820604 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1811975424}
I20240506 15:14:37.817062 22163 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
W20240506 15:14:37.843326 22241 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20240506 15:14:37.844394 22224 consensus_queue.cc:237] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.846186 22208 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.853161 22185 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.816371 22194 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240506 15:14:37.858212 22238 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240506 15:14:37.824478 21970 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:37.816128 22004 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240506 15:14:37.854202 22196 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:37.862727 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 801b36bb14184241921c1b97377314fb, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.870388 21032 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240506 15:14:37.869521 21031 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240506 15:14:37.882349 21026 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
W20240506 15:14:37.922755 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 801b36bb14184241921c1b97377314fb, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:37.929172 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 309415us
I20240506 15:14:37.930639 20996 leader_election.cc:398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.931100 20996 leader_election.cc:401] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:37.871371 22103 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:37.872483 22163 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
W20240506 15:14:37.941942 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 236725us
I20240506 15:14:37.837481 22218 consensus_queue.cc:237] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:37.920132 22183 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:37.945760 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 147159us
I20240506 15:14:37.951840 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:37.956678 20996 leader_election.cc:398] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.957973 20996 leader_election.cc:398] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:37.958359 20996 leader_election.cc:401] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20240506 15:14:37.960559 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 478260us
W20240506 15:14:37.951041 22161 consensus_meta.cc:287] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.525s user 0.011s sys 0.002s
I20240506 15:14:37.984664 21031 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.961666 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.938998 22103 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:37.994513 20846 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.995690 20846 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:37.985900 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:37.997036 21028 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:37.982939 22004 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
W20240506 15:14:37.998889 22258 log.cc:924] Time spent T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240506 15:14:37.962700 22227 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:37.941866 21970 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.006846 21970 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:38.007681 22192 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:38.008463 22192 consensus_queue.cc:1046] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:37.992144 22164 consensus_queue.cc:1046] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20240506 15:14:38.017486 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 59679.8us
I20240506 15:14:38.025034 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:38.033226 22192 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:38.034140 22071 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:38.034812 22164 consensus_queue.cc:1225] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer 7610624a16b44896b6aacea1928ab20e log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240506 15:14:38.035357 22264 raft_consensus.cc:3032] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:38.035742 22264 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:38.035928 22198 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.036615 22264 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.037279 22198 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:38.038569 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ab1516eeb0ab46f4b471c209c399130f, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:38.038928 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ab1516eeb0ab46f4b471c209c399130f, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.042299 20996 leader_election.cc:398] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:38.042891 20996 leader_election.cc:401] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:38.008790 22159 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:38.048293 21027 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.023536 22227 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5.
I20240506 15:14:38.043933 22161 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240506 15:14:38.053718 22177 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:38.054880 22183 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.049065 22264 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:38.047359 22071 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:38.017007 22004 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:38.055419 22103 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:38.042528 22220 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:38.063374 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:38.064172 21027 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:38.066927 22212 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.013s sys 0.015s
I20240506 15:14:38.069018 21028 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240506 15:14:38.069099 22212 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s user 0.014s sys 0.016s
I20240506 15:14:38.070312 21032 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:38.066219 20996 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:38.071089 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:38.069132 22227 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:38.062199 22185 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.080577 20996 leader_election.cc:398] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:38.081012 20996 leader_election.cc:401] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:38.078258 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:38.088703 22004 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.093953 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.082851 21025 raft_consensus.cc:1261] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.082841 22159 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:38.090723 22232 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 2. Current term is 3.
I20240506 15:14:38.110049 22191 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:38.040102 22230 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:38.130713 20882 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.131551 22263 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.140940 22263 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:38.146191 22246 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:38.147739 20958 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240506 15:14:38.128599 22159 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:38.154506 20880 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240506 15:14:38.083097 22071 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:38.165134 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 169463us
I20240506 15:14:38.167204 20950 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:38.167627 20846 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.154080 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:38.170627 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.172984 20880 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:38.173380 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.179265 20846 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.180976 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.185117 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.187232 22254 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.189160 22185 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.189353 22262 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240506 15:14:38.178392 22230 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.192045 21960 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:38.186026 22183 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.178637 22208 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:38.196700 22055 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:14:38.199527 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.151965 22263 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:38.210824 20880 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:38.218304 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:38.205461 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 921) took 1525 ms. Trace:
I20240506 15:14:38.231078 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:38.233821 20957 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.188058 22246 consensus_queue.cc:1046] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240506 15:14:38.237385 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:38.245713 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:38.219010 22185 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.195470 22238 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20240506 15:14:38.224016 22230 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:38.250247 20952 raft_consensus.cc:1261] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.250120 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:38.250914 20957 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:38.253270 21030 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.257380 21027 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:38.235831 22192 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:38.266659 21029 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.224794 22196 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20240506 15:14:38.267220 22266 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:38.244290 22208 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.265235 22254 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.280465 20949 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.286952 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.281661 21027 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.287938 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.284077 20956 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.290841 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.290803 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.289008 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.292749 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.293156 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:38.293835 20876 raft_consensus.cc:2362] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:38.275813 22230 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.222781 20955 rpcz_store.cc:276] 0506 15:14:36.671928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.703845 (+ 31917us) service_pool.cc:225] Handling call
0506 15:14:38.197156 (+1493311us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540033f9d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:38.197317 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1468231040}
I20240506 15:14:38.294773 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.293457 20880 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.236320 22159 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.298287 20951 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:38.299366 22192 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.296991 22266 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.290357 22185 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.507350 22192 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:38.299724 22262 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.508226 22185 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.302459 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:38.284628 22119 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.009s sys 0.004s
I20240506 15:14:38.284761 22069 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:38.517376 22119 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.808s user 0.018s sys 0.014s
I20240506 15:14:38.446480 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 51) took 1677 ms. Trace:
I20240506 15:14:38.517793 20877 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.519490 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:38.521550 22257 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.450s user 0.002s sys 0.000s
I20240506 15:14:38.523023 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.524082 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.524413 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:38.524812 20955 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:38.521123 20880 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:38.290140 22263 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:38.520210 20878 rpcz_store.cc:276] 0506 15:14:36.735366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.910807 (+175441us) service_pool.cc:225] Handling call
0506 15:14:36.944743 (+ 33936us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:37.479171 (+534428us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.412837 (+933666us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54001dae54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:38.413081 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1206859648}
W20240506 15:14:38.500658 22244 log.cc:924] Time spent T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.809s user 0.006s sys 0.000s
I20240506 15:14:38.400700 22232 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:38.518106 22269 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.057s user 0.000s sys 0.002s
I20240506 15:14:38.534981 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.516602 22196 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.531956 22263 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:38.534183 22271 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20240506 15:14:38.534577 22164 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:38.524787 20877 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.289139 22208 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.357085 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.566573 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:38.568351 20881 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.015s sys 0.066s
W20240506 15:14:38.621001 20881 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.558s user 0.015s sys 0.066s
W20240506 15:14:38.570751 22200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.001s sys 0.004s
I20240506 15:14:38.572332 20880 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:38.322084 22218 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.214s
I20240506 15:14:38.530627 21970 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20240506 15:14:38.596911 22196 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.620761 20877 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.290107 22254 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.528280 22135 consensus_queue.cc:1046] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20240506 15:14:38.509021 22262 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.634172 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.633370 20883 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:38.625422 22200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.001s sys 0.004s
I20240506 15:14:38.518718 22069 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.323674 22230 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:38.633450 22268 log.cc:924] Time spent T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20240506 15:14:38.548432 22232 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:38.620734 22238 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20240506 15:14:38.672597 20879 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.720077 20876 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:38.716725 22198 consensus_meta.cc:287] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.670s user 0.012s sys 0.000s
I20240506 15:14:38.769269 20874 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:38.761981 20883 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:38.783807 20878 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:38.731686 22187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.025s user 0.007s sys 0.010s
I20240506 15:14:38.832748 20876 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:38.745815 22185 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.880640 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:38.826304 22187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s user 0.008s sys 0.011s
W20240506 15:14:38.906231 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 225515us
I20240506 15:14:38.906162 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.908733 20880 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:38.911146 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:38.950771 20879 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:38.951424 22281 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
W20240506 15:14:38.952971 22279 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20240506 15:14:38.973681 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 330972us
I20240506 15:14:38.910061 22266 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.977941 22262 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.928504 22185 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:38.979082 22230 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.924269 22192 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:38.990949 22226 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:38.993168 22226 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:38.986614 22163 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240506 15:14:39.012387 22232 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:39.013901 22280 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240506 15:14:39.015623 22274 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20240506 15:14:39.005566 22069 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.991075 22019 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20240506 15:14:39.017292 22254 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:38.992069 22185 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.138466 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1242) took 1098 ms. Trace:
W20240506 15:14:39.139669 22086 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.002s sys 0.025s
W20240506 15:14:39.140161 22086 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.002s sys 0.025s
I20240506 15:14:39.138996 21024 rpcz_store.cc:276] 0506 15:14:38.040524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.046190 (+ 5666us) service_pool.cc:225] Handling call
0506 15:14:38.046726 (+ 536us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:38.069463 (+ 22737us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:38.069492 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:38.074438 (+ 4946us) log.cc:842] Serialized 26 byte log entry
0506 15:14:38.196914 (+122476us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b4800061ed8. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6313197431 00007f631318e527 00007f631318e3ba 00007f631328256f 00007f6313282237 00007f631329673c 00007f63132966e8 00007f6313296668 00007f6313296623 00007f63132958cc 00007f63122189c6
0506 15:14:38.308492 (+111578us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:38.308551 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.308709 (+ 158us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.138110 (+829401us) raft_consensus.cc:1671] finished
0506 15:14:39.138144 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.138353 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22264,"raft.queue_time_us":23030,"raft.run_cpu_time_us":3727,"raft.run_wall_time_us":122751,"spinlock_wait_cycles":11983744,"thread_start_us":198532,"threads_started":1,"wal-append.queue_time_us":638540}
W20240506 15:14:39.140475 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 216444us
I20240506 15:14:39.146674 22086 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.151682 22046 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.151794 22086 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.152386 22046 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.154095 22086 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.154556 22185 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.154623 22086 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.146598 22230 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.155283 22086 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.155820 22230 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.158288 22230 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.169665 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 864) took 1111 ms. Trace:
I20240506 15:14:39.146991 21026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1239) took 1096 ms. Trace:
W20240506 15:14:39.172153 22071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.000s sys 0.021s
I20240506 15:14:39.205283 22262 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.208865 22262 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.209900 22046 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.211726 22262 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.179239 21026 rpcz_store.cc:276] 0506 15:14:38.039272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.043794 (+ 4522us) service_pool.cc:225] Handling call
0506 15:14:38.080210 (+ 36416us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:38.122895 (+ 42685us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:38.128225 (+ 5330us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:38.168773 (+ 40548us) log.cc:842] Serialized 26 byte log entry
0506 15:14:38.417443 (+248670us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:38.417514 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.417656 (+ 142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.123877 (+706221us) raft_consensus.cc:1671] finished
0506 15:14:39.123925 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.130412 (+ 6487us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7786,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":11535,"thread_start_us":222239,"threads_started":1,"wal-append.queue_time_us":544200}
I20240506 15:14:39.175892 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 798) took 1492 ms. Trace:
I20240506 15:14:39.186916 22196 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:39.222010 22262 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:39.230194 22300 log.cc:924] Time spent T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.089s user 0.000s sys 0.002s
I20240506 15:14:39.223870 22208 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.269133 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:39.270848 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:39.229697 22019 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.272892 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:39.257426 22226 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.285439 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:39.267195 22086 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.263005 22292 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.273995 21960 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:39.277206 22266 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240506 15:14:39.307493 20997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.003s sys 0.015s
I20240506 15:14:39.307830 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:39.309414 20997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.675s user 0.003s sys 0.016s
I20240506 15:14:39.274711 22262 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.299379 22159 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.282408 22200 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240506 15:14:39.320183 22304 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:39.299327 22302 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.321796 22201 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.281024 22019 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.342454 22204 raft_consensus.cc:3032] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:39.342958 22204 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:39.317168 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:39.325120 22227 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.344820 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1163) took 1058 ms. Trace:
I20240506 15:14:39.174259 20881 rpcz_store.cc:276] 0506 15:14:38.058339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.058489 (+ 150us) service_pool.cc:225] Handling call
0506 15:14:38.058806 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:38.059019 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:38.910404 (+851385us) log.cc:842] Serialized 14 byte log entry
0506 15:14:39.169133 (+258729us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:39.169228 (+ 95us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:39.169314 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.169399 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.169571 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":940755,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":500,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4104,"thread_start_us":815813,"threads_started":2}
I20240506 15:14:39.321331 22114 raft_consensus.cc:3032] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:39.337201 22173 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.324146 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 806) took 1041 ms. Trace:
I20240506 15:14:39.345588 22247 raft_consensus.cc:3032] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:39.300001 22208 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:39.351943 22173 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.352670 22247 raft_consensus.cc:729] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:39.338765 22201 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.340067 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 862) took 1385 ms. Trace:
I20240506 15:14:39.353652 22247 consensus_queue.cc:260] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.354070 20875 rpcz_store.cc:276] 0506 15:14:37.954541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:37.960771 (+ 6230us) service_pool.cc:225] Handling call
0506 15:14:39.015197 (+1054426us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b58002c4998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:39.015514 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:39.016029 (+ 515us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.016080 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:39.029153 (+ 13073us) log.cc:842] Serialized 26 byte log entry
0506 15:14:39.265669 (+236516us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:39.273711 (+ 8042us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.282478 (+ 8767us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.339721 (+ 57243us) raft_consensus.cc:1671] finished
0506 15:14:39.339751 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.340003 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":133,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":3195776,"thread_start_us":129064,"threads_started":1,"wal-append.queue_time_us":256508}
I20240506 15:14:39.355542 22201 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.222746 21023 rpcz_store.cc:276] 0506 15:14:37.680673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:37.681616 (+ 943us) service_pool.cc:225] Handling call
0506 15:14:37.713674 (+ 32058us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:37.747940 (+ 34266us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:37.747978 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:38.133569 (+385591us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b2c00177f18. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630ccd1f11 00007f630cce1542 00007f630e32afc1 00007f630e2ef7da 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782
0506 15:14:38.134159 (+ 590us) log.cc:842] Serialized 26 byte log entry
0506 15:14:38.596625 (+462466us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:38.596700 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.596804 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.135472 (+538668us) raft_consensus.cc:1671] finished
0506 15:14:39.146423 (+ 10951us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.173116 (+ 26693us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2719,"raft.queue_time_us":13572,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":5059456,"thread_start_us":405294,"threads_started":1,"wal-append.queue_time_us":843007}
I20240506 15:14:39.336807 22159 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.351637 22114 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:39.329954 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1173) took 1036 ms. Trace:
I20240506 15:14:39.351296 20883 rpcz_store.cc:276] 0506 15:14:38.286585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.286719 (+ 134us) service_pool.cc:225] Handling call
0506 15:14:39.344610 (+1057891us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400267854. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:39.344742 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":795083008}
I20240506 15:14:39.341017 22302 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.352349 21029 rpcz_store.cc:276] 0506 15:14:38.283104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.283291 (+ 187us) service_pool.cc:225] Handling call
0506 15:14:38.296002 (+ 12711us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:38.296497 (+ 495us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:38.296536 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:38.301863 (+ 5327us) log.cc:842] Serialized 26 byte log entry
0506 15:14:38.626716 (+324853us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:38.626788 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.627084 (+ 296us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.323864 (+696780us) raft_consensus.cc:1671] finished
0506 15:14:39.323899 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.324073 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1521,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":61240,"thread_start_us":213409,"threads_started":1,"wal-append.queue_time_us":849719}
I20240506 15:14:39.358201 21685 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:39.358024 20876 rpcz_store.cc:276] 0506 15:14:38.294105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.294487 (+ 382us) service_pool.cc:225] Handling call
0506 15:14:39.329379 (+1034892us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54000bc554. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:39.329784 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":483319680}
I20240506 15:14:39.348520 22262 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.357542 22173 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.359990 22114 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.363445 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.365397 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.345892 22019 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.311708 22292 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.374687 20881 raft_consensus.cc:2362] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:39.375564 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1174) took 1081 ms. Trace:
I20240506 15:14:39.371452 22114 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:39.357686 22308 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240506 15:14:39.359921 22159 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:39.393293 22302 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:39.376750 20881 rpcz_store.cc:276] 0506 15:14:38.294587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.364884 (+1070297us) service_pool.cc:225] Handling call
0506 15:14:39.375481 (+ 10597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:39.389211 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:39.395202 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.397118 20881 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:39.358320 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.351881 22295 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.388561 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.396729 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:39.398154 22295 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.376540 20883 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:39.400133 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1171) took 1106 ms. Trace:
I20240506 15:14:39.344228 22204 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.361547 22227 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.401350 22163 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:39.401140 22204 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:39.401887 22163 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:39.402279 22163 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:39.403414 22227 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:39.399407 20876 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:39.405397 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1176) took 1110 ms. Trace:
I20240506 15:14:39.405750 20876 rpcz_store.cc:276] 0506 15:14:38.295278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.368377 (+1073099us) service_pool.cc:225] Handling call
0506 15:14:39.405294 (+ 36917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:39.406708 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:39.407130 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 112277us
I20240506 15:14:39.407936 22292 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:39.408224 21960 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:39.400590 20883 rpcz_store.cc:276] 0506 15:14:38.293378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.363060 (+1069682us) service_pool.cc:225] Handling call
0506 15:14:39.400055 (+ 36995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:39.409746 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.410902 21960 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.411442 21960 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.404367 22295 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.408643 20876 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:39.412725 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 45) took 1346 ms. Trace:
I20240506 15:14:39.397046 22222 raft_consensus.cc:3032] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:39.419993 20958 rpcz_store.cc:276] 0506 15:14:38.065963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.066101 (+ 138us) service_pool.cc:225] Handling call
0506 15:14:38.066387 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:38.148147 (+ 81760us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.412510 (+1264363us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b540030a554. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:39.412672 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186199296}
I20240506 15:14:39.422905 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.441082 20881 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:39.442543 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.426589 22222 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:39.444116 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.446775 20881 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:39.445073 22222 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.449220 21960 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.448010 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.451519 20881 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:39.452530 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.454867 22312 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:39.456569 22273 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.592s user 0.002s sys 0.002s
I20240506 15:14:39.447969 20876 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:39.458072 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:39.458791 20876 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:39.465068 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1240) took 1425 ms. Trace:
I20240506 15:14:39.466918 21031 rpcz_store.cc:276] 0506 15:14:38.039635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.039817 (+ 182us) service_pool.cc:225] Handling call
0506 15:14:38.051701 (+ 11884us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:38.082610 (+ 30909us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:38.082656 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:38.107583 (+ 24927us) log.cc:842] Serialized 26 byte log entry
0506 15:14:38.362026 (+254443us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:38.362065 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:38.362129 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:39.399818 (+1037689us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:39.459307 (+ 59489us) raft_consensus.cc:1671] finished
0506 15:14:39.459334 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:39.464938 (+ 5604us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24582,"raft.queue_time_us":25560,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":3854,"spinlock_wait_cycles":74990976,"thread_start_us":243172,"threads_started":1,"wal-append.queue_time_us":559361}
I20240506 15:14:39.466240 20881 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:39.471046 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.455601 22312 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.474092 20881 raft_consensus.cc:2354] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:39.445777 22204 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.476307 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.484998 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 801) took 1531 ms. Trace:
I20240506 15:14:39.477576 22312 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.478996 20954 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:39.485397 21032 rpcz_store.cc:276] 0506 15:14:37.953156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:37.953298 (+ 142us) service_pool.cc:225] Handling call
0506 15:14:37.984052 (+ 30754us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:38.074025 (+ 89973us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.483929 (+1409904us) spinlock_profiling.cc:243] Waited 86.1 ms on lock 0x7b540015e0d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:39.484941 (+ 1012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197931776}
I20240506 15:14:39.472666 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.481695 22163 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.487654 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 802) took 1255 ms. Trace:
I20240506 15:14:39.487820 20876 raft_consensus.cc:2354] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4.
I20240506 15:14:39.490577 20954 rpcz_store.cc:276] 0506 15:14:38.232394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.232860 (+ 466us) service_pool.cc:225] Handling call
0506 15:14:39.478390 (+1245530us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b58000a2998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:39.478660 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:39.487307 (+ 8647us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.487590 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9443584}
W20240506 15:14:39.500676 20924 leader_election.cc:334] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.863s (SENT)
I20240506 15:14:39.505714 20881 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20240506 15:14:39.507280 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1172) took 1213 ms. Trace:
I20240506 15:14:39.508198 20881 rpcz_store.cc:276] 0506 15:14:38.293661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.504491 (+1210830us) service_pool.cc:225] Handling call
0506 15:14:39.505243 (+ 752us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:39.506809 (+ 1566us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.507204 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:39.512434 20924 leader_election.cc:334] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.874s (SENT)
I20240506 15:14:39.516876 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:39.488513 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:39.497370 22312 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.502368 22114 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.523654 20954 raft_consensus.cc:2396] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240506 15:14:39.524235 22306 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.525103 22306 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:39.521883 20924 leader_election.cc:334] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.925s (SENT)
I20240506 15:14:39.526906 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:39.484017 22204 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:39.529107 20924 leader_election.cc:334] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.955s (SENT)
W20240506 15:14:39.530767 20924 leader_election.cc:334] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.982s (SENT)
W20240506 15:14:39.531682 20924 leader_election.cc:334] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.065s (SENT)
I20240506 15:14:39.513617 22191 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:39.533260 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 218630us
I20240506 15:14:39.513310 22192 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.534528 20953 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
W20240506 15:14:39.529104 22171 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.826s user 0.022s sys 0.020s
W20240506 15:14:39.540645 20924 leader_election.cc:334] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.098s (SENT)
I20240506 15:14:39.529286 22161 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:39.555363 20954 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:39.566185 22311 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.567153 22311 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.570883 22311 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.519513 22310 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:39.577998 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43795.8us
W20240506 15:14:39.584746 20846 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50474 dropped due to backpressure. The service queue is full; it has 50 items.
W20240506 15:14:39.551455 22171 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.848s user 0.022s sys 0.021s
I20240506 15:14:39.545593 22233 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.545240 22292 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.528947 22114 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.531240 22312 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.545121 22192 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.543717 22191 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.585415 22227 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20240506 15:14:39.867419 20874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1166) took 1577 ms (client timeout 1982 ms). Trace:
I20240506 15:14:39.820784 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:39.884297 20924 leader_election.cc:334] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.124s (SENT)
I20240506 15:14:39.885622 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.902865 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:39.903940 21030 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:39.904454 20924 leader_election.cc:334] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.183s (SENT)
W20240506 15:14:39.905280 20924 leader_election.cc:334] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.195s (SENT)
I20240506 15:14:39.905236 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.906267 21030 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:39.906116 22312 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:39.906522 20924 leader_election.cc:334] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.222s (SENT)
I20240506 15:14:39.907505 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.908504 21030 raft_consensus.cc:2379] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:39.909724 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.883286 20845 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49818 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 878)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1225)
I20240506 15:14:39.909866 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:39.913273 21970 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:39.913708 21970 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.914085 21970 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:39.914024 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:39.600363 22308 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.315s
I20240506 15:14:39.917090 22271 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:39.917536 22271 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:39.917902 22271 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:39.918402 22114 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.918689 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.615160 22310 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.676955 22208 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.921272 22208 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:39.922013 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.703450 22163 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.903034 20880 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:39.902585 20874 rpcz_store.cc:269] 0506 15:14:38.290461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.290557 (+ 96us) service_pool.cc:225] Handling call
0506 15:14:39.867169 (+1576612us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400180bd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:39.867337 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1046137600}
I20240506 15:14:39.669036 22167 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.625864 22261 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.576839 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:39.944718 20955 raft_consensus.cc:2354] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5.
I20240506 15:14:39.924702 21030 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:39.867241 22306 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.946652 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.877717 22231 raft_consensus.cc:3032] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:39.806252 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:39.857504 22204 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.815661 22222 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:39.821745 22287 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.930078 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:39.977072 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 47214.9us
I20240506 15:14:39.978302 20924 leader_election.cc:398] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4.
I20240506 15:14:39.980587 22298 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.981216 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:39.927464 22246 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.626629 22201 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.961550 21030 raft_consensus.cc:2379] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:39.823726 22188 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.818217 22103 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.811570 22295 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:40.005120 22103 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.005342 22295 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.007241 22103 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.812837 22286 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.800161 22266 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:39.733909 22193 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.943796 22310 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:39.609993 20846 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50474 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 878)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1225)
I20240506 15:14:39.981235 22298 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.973030 22261 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.024922 20846 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50474 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 878)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1222)
I20240506 15:14:39.878489 20875 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:39.868903 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.927305 22264 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:39.982106 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 50627.6us
W20240506 15:14:39.928002 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1175) took 1633 ms (client timeout 1955 ms). Trace:
W20240506 15:14:40.077788 20880 rpcz_store.cc:269] 0506 15:14:38.294998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.820396 (+1525398us) service_pool.cc:225] Handling call
0506 15:14:39.927925 (+107529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:39.924211 22208 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:39.943645 22114 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.995545 20956 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:39.969882 22290 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.999249 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.000509 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.088685 20952 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:40.089958 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.000728 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.090608 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:39.959344 22231 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:40.003149 20955 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:40.003082 22317 consensus_queue.cc:1035] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240506 15:14:40.004128 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:39.857358 22069 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:39.981477 21685 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:39.989681 22306 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:39.929090 22292 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:39.599426 22315 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:39.968989 20845 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:40.109689 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 140765us
I20240506 15:14:40.101991 21030 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:40.039885 20846 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50474 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 878)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 884)
I20240506 15:14:40.076360 22186 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.013111 22188 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.113364 20846 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 878)
I20240506 15:14:39.931787 22167 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.052790 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:39.929044 22233 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:40.086409 21032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.015s sys 0.048s
I20240506 15:14:40.092545 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.096596 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.096696 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.091439 20880 raft_consensus.cc:2362] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:40.099578 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.056846 22302 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.126873 20952 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:40.134326 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.112075 20845 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
W20240506 15:14:40.125859 21032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.015s sys 0.048s
I20240506 15:14:40.009703 22201 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.127849 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.093223 22298 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.142026 20949 raft_consensus.cc:1261] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:40.089707 22295 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.137418 20880 raft_consensus.cc:2362] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:40.131837 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.146522 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.135768 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:40.148469 20952 raft_consensus.cc:2396] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:40.139432 20957 raft_consensus.cc:1226] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:40.061239 22286 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.152422 20880 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:40.023926 22266 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.050753 22193 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:40.169102 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 116402us
I20240506 15:14:40.171283 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.180594 21030 raft_consensus.cc:2379] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:40.094575 22290 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.183014 22233 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.183473 22290 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.183909 22295 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:40.153198 22207 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:40.184484 22295 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:40.185050 22295 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:40.188742 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:40.131572 22208 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240506 15:14:40.124080 22314 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.538s user 0.012s sys 0.034s
I20240506 15:14:40.095948 22114 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.002175 22287 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.098484 22212 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20240506 15:14:40.131330 20882 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:40.001510 22246 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.163951 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:40.166271 20954 consensus_meta.cc:287] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.607s user 0.011s sys 0.000s
I20240506 15:14:40.133283 22159 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20240506 15:14:40.114743 22069 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.103506 22271 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.132819 22186 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.153352 22315 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:40.514252 20921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.131:42913 timed out after 1.525s (SENT) [suppressed 60 similar messages]
W20240506 15:14:40.717121 20998 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.017s sys 0.040s
W20240506 15:14:40.360361 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 282280us
I20240506 15:14:40.805562 20880 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:40.806619 20998 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.020s sys 0.043s
W20240506 15:14:40.809319 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226) took 1270 ms (client timeout 1674 ms). Trace:
W20240506 15:14:40.810567 20880 rpcz_store.cc:269] 0506 15:14:39.538661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.163625 (+624964us) service_pool.cc:225] Handling call
0506 15:14:40.809255 (+645630us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.160825 22263 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.238235 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.812126 20954 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:40.812721 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.813689 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 49) took 1280 ms. Trace:
W20240506 15:14:40.766878 22222 consensus_meta.cc:287] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.783s user 0.012s sys 0.001s
W20240506 15:14:40.440937 22325 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
I20240506 15:14:40.816908 20954 rpcz_store.cc:276] 0506 15:14:39.533209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.533332 (+ 123us) service_pool.cc:225] Handling call
0506 15:14:39.554203 (+ 20871us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:40.813423 (+1259220us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:40.813634 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.401926 22316 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:40.316617 22319 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.222s user 0.002s sys 0.000s
I20240506 15:14:40.266047 21654 leader_election.cc:398] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5.
I20240506 15:14:40.812286 22333 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:40.674655 21685 consensus_meta.cc:287] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.544s user 0.008s sys 0.001s
I20240506 15:14:40.820204 21030 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:40.389268 22185 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.218714 22208 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:40.727018 22324 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.326s user 0.000s sys 0.000s
I20240506 15:14:40.828919 21654 leader_election.cc:401] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20240506 15:14:40.822710 20957 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.008s sys 0.017s
W20240506 15:14:40.840390 20957 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.009s sys 0.017s
I20240506 15:14:40.273798 22290 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.686038 22207 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.218362 22191 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.729424 22167 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.728889 22292 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.850008 22207 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.853487 22191 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:40.854254 22207 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.855134 22207 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.858317 20953 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240506 15:14:40.732975 22201 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.724097 22188 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.273732 22192 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.728014 22193 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.729414 22286 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.734174 22266 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.155169 22302 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.317322 22261 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.560761 22019 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:40.359148 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 426771us
W20240506 15:14:40.828646 21970 consensus_meta.cc:287] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.914s user 0.010s sys 0.003s
W20240506 15:14:40.820614 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1190) took 2183 ms (client timeout 1711 ms). Trace:
I20240506 15:14:40.838395 20880 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:40.882084 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227) took 1343 ms (client timeout 1687 ms). Trace:
W20240506 15:14:40.882488 20880 rpcz_store.cc:269] 0506 15:14:39.538909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.811515 (+1272606us) service_pool.cc:225] Handling call
0506 15:14:40.882025 (+ 70510us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.842095 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1250) took 1306 ms. Trace:
I20240506 15:14:40.842379 22186 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.835222 21685 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:40.839661 22185 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:40.846132 20848 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.047s user 0.023s sys 0.064s
W20240506 15:14:40.891391 20848 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.092s user 0.023s sys 0.064s
I20240506 15:14:40.403316 22295 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.839790 22208 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:40.829452 22316 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.886409 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.839478 22159 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:40.881865 20882 rpcz_store.cc:269] 0506 15:14:38.635163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.868402 (+1233239us) service_pool.cc:225] Handling call
0506 15:14:40.817942 (+949540us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:40.896788 22256 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 2.727s user 0.007s sys 0.020s
I20240506 15:14:40.896901 22069 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.897470 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.861510 22207 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.898134 20882 raft_consensus.cc:2362] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
W20240506 15:14:40.898703 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229) took 1359 ms (client timeout 1724 ms). Trace:
W20240506 15:14:40.899099 20882 rpcz_store.cc:269] 0506 15:14:39.539447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.897169 (+1357722us) service_pool.cc:225] Handling call
0506 15:14:40.898650 (+ 1481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.899550 22207 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.883273 21032 rpcz_store.cc:276] 0506 15:14:39.535993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.536146 (+ 153us) service_pool.cc:225] Handling call
0506 15:14:39.536506 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:39.536707 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.536722 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:39.544942 (+ 8220us) log.cc:842] Serialized 26 byte log entry
0506 15:14:40.135800 (+590858us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:40.135861 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:40.135963 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:40.841848 (+705885us) raft_consensus.cc:1671] finished
0506 15:14:40.841873 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:40.842026 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":558926,"threads_started":1,"wal-append.queue_time_us":599070}
I20240506 15:14:40.887159 21685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34002 (request call id 68) took 1198 ms. Trace:
W20240506 15:14:40.877632 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 699656us
I20240506 15:14:40.893100 20880 raft_consensus.cc:2379] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:40.900558 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:40.928241 22230 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:40.928347 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228) took 1389 ms (client timeout 1711 ms). Trace:
W20240506 15:14:40.928858 20880 rpcz_store.cc:269] 0506 15:14:39.539181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.886147 (+1346966us) service_pool.cc:225] Handling call
0506 15:14:40.928284 (+ 42137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.928926 22230 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.901825 22261 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.929911 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.930403 22261 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.883842 22019 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:40.930701 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230) took 1391 ms (client timeout 1736 ms). Trace:
W20240506 15:14:40.931869 20880 rpcz_store.cc:269] 0506 15:14:39.539704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.929654 (+1389950us) service_pool.cc:225] Handling call
0506 15:14:40.930649 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.895273 22159 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.901638 22192 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.932893 22192 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.933512 22019 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.933943 22159 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.895962 22186 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:40.900197 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:40.934792 22186 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.935271 22192 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.901436 22331 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.936331 22331 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.936812 22186 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.937362 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.938302 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:40.938983 22331 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.889266 22271 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.908182 22188 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.883612 21970 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:40.940659 22188 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.834988 22333 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.914819 22246 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.904338 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1255) took 1366 ms. Trace:
I20240506 15:14:40.915571 22315 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:40.935570 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232) took 1395 ms (client timeout 1726 ms). Trace:
I20240506 15:14:40.905083 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 63) took 2401 ms. Trace:
I20240506 15:14:40.942667 20958 raft_consensus.cc:2379] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:40.939023 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.942044 21970 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:40.848162 22290 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:40.965921 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 158618us
I20240506 15:14:40.892920 22212 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5.
I20240506 15:14:40.970153 21032 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:40.911326 22185 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.926098 21685 rpcz_store.cc:276] 0506 15:14:39.688590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.732443 (+ 43853us) service_pool.cc:225] Handling call
0506 15:14:40.885931 (+1153488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:40.966217 21029 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:40.970688 20876 rpcz_store.cc:276] 0506 15:14:38.504379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.503063 (+998684us) service_pool.cc:225] Handling call
0506 15:14:39.503584 (+ 521us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:39.510187 (+ 6603us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.510221 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:14:39.513571 (+ 3350us) log.cc:842] Serialized 52 byte log entry
0506 15:14:39.934150 (+420579us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:14:39.934214 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.934323 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:40.904814 (+970491us) raft_consensus.cc:1671] finished
0506 15:14:40.904841 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:40.905017 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4711,"raft.queue_time_us":6387,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"thread_start_us":400707,"threads_started":1,"wal-append.queue_time_us":572695}
I20240506 15:14:40.972478 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:40.979951 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:40.973798 22333 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.976038 20996 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:40.960065 21024 rpcz_store.cc:276] 0506 15:14:39.538064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.539597 (+ 1533us) service_pool.cc:225] Handling call
0506 15:14:39.543106 (+ 3509us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:39.543421 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.543491 (+ 70us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:39.880494 (+337003us) log.cc:842] Serialized 26 byte log entry
0506 15:14:40.319073 (+438579us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:40.319132 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:40.319238 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:40.903998 (+584760us) raft_consensus.cc:1671] finished
0506 15:14:40.904027 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:40.904217 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":437959,"threads_started":1,"wal-append.queue_time_us":989953}
I20240506 15:14:40.978680 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1261) took 1101 ms. Trace:
I20240506 15:14:40.982142 21032 rpcz_store.cc:276] 0506 15:14:39.877520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.938669 (+1061149us) service_pool.cc:225] Handling call
0506 15:14:40.978580 (+ 39911us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:40.974934 22330 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20240506 15:14:40.980629 20876 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:40.983724 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.920015 22316 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.986824 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220) took 1450 ms. Trace:
I20240506 15:14:40.990218 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:40.992274 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.993000 21032 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:40.989785 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 475510us
I20240506 15:14:40.990422 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1251) took 1454 ms. Trace:
I20240506 15:14:40.994714 22290 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.995043 21970 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:40.995532 21970 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:40.993839 21031 rpcz_store.cc:276] 0506 15:14:39.536455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.536596 (+ 141us) service_pool.cc:225] Handling call
0506 15:14:40.841861 (+1305265us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:40.842092 (+ 231us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:40.842443 (+ 351us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:40.842474 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:40.842962 (+ 488us) log.cc:842] Serialized 26 byte log entry
0506 15:14:40.975828 (+132866us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:40.975865 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:40.975925 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:40.985934 (+ 10009us) raft_consensus.cc:1671] finished
0506 15:14:40.985952 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:40.990335 (+ 4383us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":620,"raft.queue_time_us":14552,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":2834176,"thread_start_us":132461,"threads_started":1,"wal-append.queue_time_us":140601}
I20240506 15:14:40.996367 22302 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:40.991500 22114 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.997618 20957 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:40.998358 22295 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.942059 22208 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:40.999554 22188 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.987334 22261 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:41.000090 22208 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.991550 22287 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:40.991290 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:40.990571 20876 rpcz_store.cc:276] 0506 15:14:39.536085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.979660 (+1443575us) service_pool.cc:225] Handling call
0506 15:14:40.986770 (+ 7110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.001822 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 812) took 1447 ms. Trace:
I20240506 15:14:41.002308 21024 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:40.991917 20880 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:41.011797 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:40.996397 21970 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:41.018970 22106 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:40.990403 22333 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:41.034948 22208 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.057977 22333 raft_consensus.cc:686] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:41.058746 22208 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.059243 22333 consensus_queue.cc:237] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.061594 22208 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:41.035640 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:41.062913 22159 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.063544 22159 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.037397 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:41.065012 22186 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.063140 21031 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:41.065958 22186 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.066476 22159 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.067492 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:41.039909 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.068609 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1270) took 1188 ms. Trace:
I20240506 15:14:41.069036 21031 rpcz_store.cc:276] 0506 15:14:39.903638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.067099 (+1163461us) service_pool.cc:225] Handling call
0506 15:14:41.068546 (+ 1447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.069775 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.070526 20957 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:41.071595 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.072171 20957 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:41.073320 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.073536 22247 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:41.002367 21032 rpcz_store.cc:276] 0506 15:14:39.554616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.992014 (+1437398us) service_pool.cc:225] Handling call
0506 15:14:41.001765 (+ 9751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.075816 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:41.040724 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.042625 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:41.032168 21024 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:41.078253 21690 raft_consensus.cc:2379] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:41.078370 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1272) took 1115 ms. Trace:
I20240506 15:14:41.078974 21024 rpcz_store.cc:276] 0506 15:14:39.963364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.991050 (+1027686us) service_pool.cc:225] Handling call
0506 15:14:41.078299 (+ 87249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.048215 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:41.041554 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231) took 1501 ms (client timeout 1739 ms). Trace:
I20240506 15:14:41.041096 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:41.046952 20998 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47952 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 6 similar messages]
I20240506 15:14:41.042095 20876 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
W20240506 15:14:41.086282 20924 leader_election.cc:334] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.549s (SENT)
W20240506 15:14:41.081950 20880 rpcz_store.cc:269] 0506 15:14:39.539969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.937897 (+1397928us) service_pool.cc:225] Handling call
0506 15:14:41.041496 (+103599us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:41.092759 20924 leader_election.cc:334] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.555s (SENT)
I20240506 15:14:41.080209 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:41.077010 20953 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:41.095820 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 823) took 1271 ms. Trace:
I20240506 15:14:41.092664 20954 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:41.096542 21031 rpcz_store.cc:276] 0506 15:14:39.824915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.069914 (+1244999us) service_pool.cc:225] Handling call
0506 15:14:41.095750 (+ 25836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.097919 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1278) took 1076 ms. Trace:
I20240506 15:14:41.098362 21024 rpcz_store.cc:276] 0506 15:14:40.021196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.079880 (+1058684us) service_pool.cc:225] Handling call
0506 15:14:41.097853 (+ 17973us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.075974 20845 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49818 (request call id 70)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71)
W20240506 15:14:41.103546 20924 leader_election.cc:334] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.567s (SENT)
I20240506 15:14:41.104266 20924 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:40.974706 20882 rpcz_store.cc:269] 0506 15:14:39.540216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.899924 (+1359708us) service_pool.cc:225] Handling call
0506 15:14:40.935506 (+ 35582us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.047859 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:41.109663 20953 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:41.111980 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.106647 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.047488 22340 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.047744 22246 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:41.095731 22159 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.044903 22316 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:41.045177 22290 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.108605 22019 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240506 15:14:41.164557 20924 leader_election.cc:334] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.628s (SENT)
I20240506 15:14:41.112848 20953 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:41.041671 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:41.165082 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:41.086910 20998 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47952 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 825)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1268)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 837)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 80)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 836)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 840)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 842)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 75)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 79)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 818)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1266)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1276)
I20240506 15:14:41.124483 22186 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.125030 22340 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.126610 20957 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:41.107975 22332 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.109794 22220 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:41.111846 22292 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.108711 22302 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.129845 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 886) took 1018 ms. Trace:
I20240506 15:14:41.109918 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:41.097190 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 102669us
I20240506 15:14:41.169477 22316 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.167784 22290 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.157939 22246 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.703505 22350 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.704100 22350 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:41.705363 20877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1184) took 3254 ms (client timeout 1894 ms). Trace:
W20240506 15:14:41.705812 20877 rpcz_store.cc:269] 0506 15:14:38.450729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.705143 (+3254414us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:41.705274 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:41.706650 22350 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:41.706816 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.707589 20877 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:41.166008 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.708544 20953 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:41.709378 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:41.170091 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1181) took 2861 ms (client timeout 1736 ms). Trace:
I20240506 15:14:41.710317 22294 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240506 15:14:41.710594 20878 rpcz_store.cc:269] 0506 15:14:38.296581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.528510 (+231929us) service_pool.cc:225] Handling call
0506 15:14:41.157925 (+2629415us) spinlock_profiling.cc:243] Waited 59.9 ms on lock 0x7b54002aab54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:41.158069 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":137656192}
I20240506 15:14:41.711665 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.712116 22340 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.712282 20878 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
W20240506 15:14:41.712782 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236) took 1683 ms (client timeout 2014 ms). Trace:
I20240506 15:14:41.687261 22292 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:41.713116 20878 rpcz_store.cc:269] 0506 15:14:40.029057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.711408 (+1682351us) service_pool.cc:225] Handling call
0506 15:14:41.712731 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.713985 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:41.714643 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883) took 2189 ms (client timeout 2564 ms). Trace:
I20240506 15:14:41.686482 20882 rpcz_store.cc:276] 0506 15:14:40.111528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.106319 (+994791us) service_pool.cc:225] Handling call
0506 15:14:41.129784 (+ 23465us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:41.715068 20878 rpcz_store.cc:269] 0506 15:14:39.525516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.713733 (+2188217us) service_pool.cc:225] Handling call
0506 15:14:41.714598 (+ 865us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.715624 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.158732 22159 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:41.716435 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882) took 2195 ms (client timeout 2630 ms). Trace:
W20240506 15:14:41.716794 20882 rpcz_store.cc:269] 0506 15:14:39.520865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.715378 (+2194513us) service_pool.cc:225] Handling call
0506 15:14:41.716381 (+ 1003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.717785 22159 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.718276 22159 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:41.718737 22159 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:41.688331 22302 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.094669 22106 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.721446 22302 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.721545 22106 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.582233 22331 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.582933 22271 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:41.676844 22220 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.722251 22331 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.722424 22271 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.723809 22220 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.725097 22286 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:41.725240 22271 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:41.725459 22106 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:41.249104 22347 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.205s
I20240506 15:14:41.726068 22106 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.726648 22351 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:41.443030 22289 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.727186 22351 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:41.486745 20883 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:41.664081 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:41.708084 20877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880) took 2190 ms (client timeout 2416 ms). Trace:
I20240506 15:14:41.668892 22183 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.675602 22332 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.664384 22247 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:41.698261 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.746305 20952 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:41.698988 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:41.700497 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1245) took 3465 ms. Trace:
I20240506 15:14:41.700935 22316 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.702677 22290 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.725287 22246 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.729151 20953 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
W20240506 15:14:41.839306 22326 log.cc:924] Time spent T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 1.892s user 0.006s sys 0.038s
I20240506 15:14:41.730654 22287 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:41.840096 20958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35816 (request call id 806) took 1859 ms (client timeout 1664 ms). Trace:
I20240506 15:14:41.728235 22106 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.840030 22287 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.840535 20951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 53) took 2305 ms. Trace:
I20240506 15:14:41.715911 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.841121 20955 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:41.841627 20878 raft_consensus.cc:2396] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240506 15:14:41.840569 20958 rpcz_store.cc:269] 0506 15:14:39.980750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.980916 (+ 166us) service_pool.cc:225] Handling call
0506 15:14:41.839893 (+1858977us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b5400305f54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:41.840038 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2208748928}
W20240506 15:14:41.842340 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235) took 1818 ms (client timeout 2304 ms). Trace:
I20240506 15:14:41.842778 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 54) took 1738 ms. Trace:
W20240506 15:14:41.843600 22337 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.997s user 0.000s sys 0.026s
I20240506 15:14:41.840987 20951 rpcz_store.cc:276] 0506 15:14:39.535373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.543058 (+ 7685us) service_pool.cc:225] Handling call
0506 15:14:39.644625 (+101567us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:39.684486 (+ 39861us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.687609 (+ 3123us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:39.869401 (+181792us) log.cc:842] Serialized 26 byte log entry
0506 15:14:39.946568 (+ 77167us) raft_consensus.cc:1631] Marking committed up to 6
0506 15:14:39.946637 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:39.946749 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:41.838791 (+1892042us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:41.840162 (+ 1371us) raft_consensus.cc:1671] finished
0506 15:14:41.840206 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:41.840451 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51281,"raft.queue_time_us":165,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":5576,"spinlock_wait_cycles":156471040,"thread_start_us":76795,"threads_started":1,"wal-append.queue_time_us":77207}
I20240506 15:14:41.846710 22332 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.847470 22247 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.850371 22247 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:41.749816 22183 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:41.854419 22231 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:41.714664 22292 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:41.732858 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:41.729324 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 65340.5us
I20240506 15:14:41.727629 22289 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:41.732712 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:41.717685 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:41.757124 20874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 881) took 2238 ms (client timeout 1550 ms). Trace:
I20240506 15:14:41.759832 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 174) took 6383 ms. Trace:
I20240506 15:14:41.762182 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.765084 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 870) took 3253 ms. Trace:
I20240506 15:14:41.731356 22351 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:41.057608 20997 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47948 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1268)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 837)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 80)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 836)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 840)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 842)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 75)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 79)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 818)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1266)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1276)
W20240506 15:14:41.781581 22278 log.cc:924] Time spent T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 2.999s user 0.017s sys 0.031s
I20240506 15:14:41.786093 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 193) took 4905 ms. Trace:
I20240506 15:14:40.781478 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240506 15:14:41.789005 20883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1192) took 3064 ms (client timeout 2082 ms). Trace:
I20240506 15:14:41.788538 20779 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 5 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:41.790792 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1252) took 2254 ms. Trace:
I20240506 15:14:41.792626 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:41.795104 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:40.116520 20846 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 885)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
I20240506 15:14:41.747763 21028 rpcz_store.cc:276] 0506 15:14:38.234785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.237012 (+ 2227us) service_pool.cc:225] Handling call
0506 15:14:41.048661 (+2811649us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:41.048876 (+ 215us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:41.049299 (+ 423us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:41.049333 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:41.049849 (+ 516us) log.cc:842] Serialized 26 byte log entry
0506 15:14:41.500173 (+450324us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:41.500234 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:41.509343 (+ 9109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:41.700089 (+190746us) raft_consensus.cc:1671] finished
0506 15:14:41.700119 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:41.700421 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3688,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":5442,"spinlock_wait_cycles":2795904,"thread_start_us":241676,"threads_started":1,"wal-append.queue_time_us":648332}
I20240506 15:14:41.744395 22271 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240506 15:14:41.729025 20877 rpcz_store.cc:269] 0506 15:14:39.517904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.706497 (+2188593us) service_pool.cc:225] Handling call
0506 15:14:41.708028 (+ 1531us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:41.842444 22318 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 1.895s user 0.011s sys 0.029s
W20240506 15:14:41.774269 20875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1188) took 3241 ms (client timeout 1925 ms). Trace:
I20240506 15:14:41.856812 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:41.858100 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 826) took 1916 ms (client timeout 1976 ms). Trace:
I20240506 15:14:41.859329 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:41.859861 20874 rpcz_store.cc:269] 0506 15:14:39.518828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.756795 (+2237967us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:41.756988 (+ 193us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:41.842794 20878 rpcz_store.cc:269] 0506 15:14:40.024303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.715691 (+1691388us) service_pool.cc:225] Handling call
0506 15:14:41.842278 (+126587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.860627 20876 raft_consensus.cc:2396] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240506 15:14:42.094722 22159 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:42.142733 22346 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.001s sys 0.017s
I20240506 15:14:42.179440 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:42.182032 21018 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 with delete_type TABLET_DATA_TOMBSTONED (TS 7610624a16b44896b6aacea1928ab20e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47936
W20240506 15:14:42.093549 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1234) took 2052 ms (client timeout 2474 ms). Trace:
W20240506 15:14:41.974246 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 299183us
W20240506 15:14:42.184748 20922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.019s sys 0.017s
W20240506 15:14:42.184990 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 282978us
I20240506 15:14:41.895781 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:42.180702 20876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 875) took 2666 ms (client timeout 2288 ms). Trace:
W20240506 15:14:42.183382 20883 rpcz_store.cc:269] 0506 15:14:38.727143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.422561 (+695418us) service_pool.cc:225] Handling call
0506 15:14:41.788816 (+2366255us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400267354. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:41.788935 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4549259008}
I20240506 15:14:42.228547 20880 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:41.860798 20881 rpcz_store.cc:276] 0506 15:14:38.511662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.513632 (+1001970us) service_pool.cc:225] Handling call
0506 15:14:39.513955 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:39.953837 (+439882us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:39.953883 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:39.953949 (+ 66us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:39.954007 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:41.764765 (+1810758us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54000fbc54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:41.764838 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:41.765008 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1010868608}
W20240506 15:14:42.184546 21032 rpcz_store.cc:269] 0506 15:14:39.941344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.074766 (+1133422us) service_pool.cc:225] Handling call
0506 15:14:41.732730 (+657964us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:41.858017 (+125287us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3166080}
W20240506 15:14:42.230051 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1238) took 2113 ms (client timeout 1759 ms). Trace:
W20240506 15:14:42.228788 20922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.019s sys 0.024s
W20240506 15:14:42.228349 20876 rpcz_store.cc:269] 0506 15:14:39.513976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.761945 (+2247969us) service_pool.cc:225] Handling call
0506 15:14:42.180635 (+418690us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:41.868607 20950 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:42.142915 21970 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.008s sys 0.012s
W20240506 15:14:42.231735 20880 rpcz_store.cc:269] 0506 15:14:40.116136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.856546 (+1740410us) service_pool.cc:225] Handling call
0506 15:14:42.229982 (+373436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:42.209137 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:41.856158 22183 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:42.320964 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 815) took 2764 ms (client timeout 2560 ms). Trace:
I20240506 15:14:42.944875 22183 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:42.207346 21031 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
W20240506 15:14:42.320612 21028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1262) took 2425 ms (client timeout 2181 ms). Trace:
W20240506 15:14:42.947055 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 825) took 3012 ms (client timeout 2083 ms). Trace:
W20240506 15:14:42.284840 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 56393.9us
W20240506 15:14:42.947825 21031 rpcz_store.cc:269] 0506 15:14:39.934568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.792348 (+1857780us) service_pool.cc:225] Handling call
0506 15:14:42.946945 (+1154597us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:42.763222 21027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.006s sys 0.030s
W20240506 15:14:42.949276 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1269) took 3069 ms (client timeout 2304 ms). Trace:
W20240506 15:14:42.949414 21027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s user 0.006s sys 0.034s
W20240506 15:14:42.949697 21031 rpcz_store.cc:269] 0506 15:14:39.880006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.949066 (+3069060us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.949186 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:42.950600 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 817) took 3394 ms (client timeout 2632 ms). Trace:
W20240506 15:14:42.219334 20924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.017s sys 0.019s
W20240506 15:14:42.319989 20954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.010s sys 0.022s
W20240506 15:14:42.951761 20924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.269s user 0.023s sys 0.032s
W20240506 15:14:42.952718 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 1.78786e+06us
I20240506 15:14:41.860185 20776 rpcz_store.cc:276] 0506 15:14:35.376574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:35.376788 (+ 214us) service_pool.cc:225] Handling call
0506 15:14:41.759743 (+6382955us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:36.620701 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:36.733975 (+113274us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:36.736503 (+ 2528us) write_op.cc:435] Acquired schema lock
0506 15:14:36.738971 (+ 2468us) tablet.cc:660] Decoding operations
0506 15:14:36.867209 (+128238us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:36.881139 (+ 13930us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:36.891342 (+ 10203us) tablet.cc:683] Acquiring locks for 2 operations
0506 15:14:36.966517 (+ 75175us) tablet.cc:699] Row locks acquired
0506 15:14:36.966550 (+ 33us) write_op.cc:260] PREPARE: finished
0506 15:14:36.975378 (+ 8828us) write_op.cc:270] Start()
0506 15:14:36.979756 (+ 4378us) write_op.cc:276] Timestamp: P: 1715008476973493 usec, L: 0
0506 15:14:36.989407 (+ 9651us) op_driver.cc:331] REPLICATION: starting
0506 15:14:37.046853 (+ 57446us) log.cc:842] Serialized 918 byte log entry
0506 15:14:37.150595 (+103742us) op_driver.cc:444] REPLICATION: finished
0506 15:14:37.200700 (+ 50105us) write_op.cc:301] APPLY: starting
0506 15:14:37.200782 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:37.200903 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:37.201026 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:37.201418 (+ 392us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:37.201623 (+ 205us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 15:14:37.201692 (+ 69us) write_op.cc:312] APPLY: finished
0506 15:14:37.201974 (+ 282us) log.cc:842] Serialized 28 byte log entry
0506 15:14:37.202164 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:37.202255 (+ 91us) write_op.cc:454] Released schema lock
0506 15:14:37.202294 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:37.202559 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:37.202589 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":49831,"apply.run_cpu_time_us":2459,"apply.run_wall_time_us":2456,"num_ops":2,"prepare.queue_time_us":31497,"prepare.run_cpu_time_us":5345,"prepare.run_wall_time_us":426521,"replication_time_us":158645,"thread_start_us":68317,"threads_started":2}]]}
I20240506 15:14:42.868264 22289 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:42.946967 21028 rpcz_store.cc:269] 0506 15:14:39.877985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.253500 (+2375515us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.303488 (+ 49988us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:42.384243 20845 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.005s sys 0.021s
W20240506 15:14:42.955943 21970 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.787s user 0.020s sys 0.024s
I20240506 15:14:42.180866 22177 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:42.955704 20845 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.086s user 0.009s sys 0.025s
W20240506 15:14:42.957180 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 1.28944e+06us
I20240506 15:14:42.957141 22177 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:42.959931 22177 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:42.184509 20780 rpcz_store.cc:276] 0506 15:14:36.880489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.896760 (+ 16271us) service_pool.cc:225] Handling call
0506 15:14:41.786018 (+4889258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:37.047485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:37.048225 (+ 740us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:37.048297 (+ 72us) write_op.cc:435] Acquired schema lock
0506 15:14:37.048324 (+ 27us) tablet.cc:660] Decoding operations
0506 15:14:37.048768 (+ 444us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:37.048838 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:37.048879 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:37.049143 (+ 264us) tablet.cc:699] Row locks acquired
0506 15:14:37.049166 (+ 23us) write_op.cc:260] PREPARE: finished
0506 15:14:37.049381 (+ 215us) write_op.cc:270] Start()
0506 15:14:37.049528 (+ 147us) write_op.cc:276] Timestamp: P: 1715008477049304 usec, L: 0
0506 15:14:37.049553 (+ 25us) op_driver.cc:331] REPLICATION: starting
0506 15:14:37.050453 (+ 900us) log.cc:842] Serialized 540 byte log entry
0506 15:14:37.200762 (+150309us) op_driver.cc:444] REPLICATION: finished
0506 15:14:37.202713 (+ 1951us) write_op.cc:301] APPLY: starting
0506 15:14:37.202805 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:37.202917 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:37.203091 (+ 174us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:37.244176 (+ 41085us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:37.268298 (+ 24122us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:37.272597 (+ 4299us) write_op.cc:312] APPLY: finished
0506 15:14:37.287737 (+ 15140us) log.cc:842] Serialized 22 byte log entry
0506 15:14:37.308666 (+ 20929us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:37.308753 (+ 87us) write_op.cc:454] Released schema lock
0506 15:14:37.308799 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:37.309073 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:37.309110 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4449822,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":1581,"apply.run_cpu_time_us":2831,"apply.run_wall_time_us":106779,"num_ops":1,"prepare.queue_time_us":55970,"prepare.run_cpu_time_us":3278,"prepare.run_wall_time_us":3277,"replication_time_us":151079,"thread_start_us":1050,"threads_started":1}]]}
W20240506 15:14:42.951817 20954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.275s user 0.030s sys 0.055s
I20240506 15:14:42.303217 20952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 973) took 1137 ms. Trace:
W20240506 15:14:42.947134 20950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35816 (request call id 797) took 4718 ms (client timeout 2976 ms). Trace:
W20240506 15:14:42.955683 21028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1268) took 3076 ms (client timeout 2474 ms). Trace:
I20240506 15:14:42.230862 21029 rpcz_store.cc:276] 0506 15:14:39.536808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.536947 (+ 139us) service_pool.cc:225] Handling call
0506 15:14:40.592897 (+1055950us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:40.593162 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:40.979273 (+386111us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:41.790524 (+811251us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b540039dad4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:41.790730 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":859557888}
W20240506 15:14:42.181759 22346 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.736s user 0.001s sys 0.017s
W20240506 15:14:42.206327 22333 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.001s sys 0.010s
I20240506 15:14:41.869524 22331 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:41.843390 20955 rpcz_store.cc:276] 0506 15:14:40.104024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.202359 (+ 98335us) service_pool.cc:225] Handling call
0506 15:14:41.840922 (+1638563us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:41.841715 (+ 793us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:41.842593 (+ 878us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400305f50. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eec59 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:41.842726 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2599887232}
W20240506 15:14:42.964558 20874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 893) took 1926 ms (client timeout 1739 ms). Trace:
I20240506 15:14:42.206923 22271 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:42.790298 21018 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.012s sys 0.010s
W20240506 15:14:42.965456 22187 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.003s sys 0.013s
I20240506 15:14:42.318382 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:42.208899 20957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 967) took 1052 ms. Trace:
I20240506 15:14:42.942765 20878 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
W20240506 15:14:43.007615 20998 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47952 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 2 similar messages]
W20240506 15:14:42.993973 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 50884.5us
I20240506 15:14:41.870741 22230 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:42.763443 21654 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.020s sys 0.020s
W20240506 15:14:43.047273 20883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 888) took 2058 ms (client timeout 1870 ms). Trace:
W20240506 15:14:43.158668 21029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 832) took 2183 ms (client timeout 1599 ms). Trace:
I20240506 15:14:42.753636 22355 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.010975 21970 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:42.983417 20921 connection.cc:232] RPC call timeout handler was delayed by -1.8857891480000717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
I20240506 15:14:43.008769 20997 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47952 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 837)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 80)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 836)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 840)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 75)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 79)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 818)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1266)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1276)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 81)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 845)
W20240506 15:14:42.945174 21032 rpcz_store.cc:269] 0506 15:14:39.555590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.285447 (+2729857us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.320126 (+ 34679us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:42.945536 22183 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:43.158053 21654 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.439s user 0.020s sys 0.025s
W20240506 15:14:42.206175 20875 rpcz_store.cc:269] 0506 15:14:38.533112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.356010 (+822898us) service_pool.cc:225] Handling call
0506 15:14:41.773948 (+2417938us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b540006e854. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:41.774147 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30579968}
I20240506 15:14:43.161170 20952 rpcz_store.cc:276] 0506 15:14:41.166046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.747305 (+581259us) service_pool.cc:225] Handling call
0506 15:14:42.303138 (+555833us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.158807 20877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 892) took 2154 ms (client timeout 1594 ms). Trace:
W20240506 15:14:43.094576 20924 leader_election.cc:334] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.674s (SENT)
W20240506 15:14:43.104888 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.518s user 0.009s sys 0.009s
W20240506 15:14:43.175654 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.589s user 0.012s sys 0.012s
W20240506 15:14:43.175879 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 81381.9us
W20240506 15:14:43.159051 21018 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.975s user 0.018s sys 0.022s
W20240506 15:14:42.632555 20879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1189) took 4012 ms (client timeout 1739 ms). Trace:
I20240506 15:14:42.385774 21023 raft_consensus.cc:2354] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
W20240506 15:14:42.986322 20845 connection.cc:232] RPC call timeout handler was delayed by -1.2736504270000069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20240506 15:14:42.206208 20882 rpcz_store.cc:269] 0506 15:14:40.023960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.717429 (+1693469us) service_pool.cc:225] Handling call
0506 15:14:42.076441 (+359012us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.159394 20883 rpcz_store.cc:269] 0506 15:14:40.988812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.964367 (+1975555us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.047194 (+ 82827us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.160156 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49818 (request call id 70) took 2270 ms (client timeout 1810 ms). Trace:
W20240506 15:14:43.162106 21028 rpcz_store.cc:269] 0506 15:14:39.879693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.955458 (+3075765us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.955605 (+ 147us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.167318 20921 connection.cc:232] RPC call timeout handler was delayed by -1.7050938690000521s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20240506 15:14:43.159011 22271 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:43.158558 22187 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.983s user 0.008s sys 0.020s
W20240506 15:14:43.168207 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 835) took 2185 ms (client timeout 1600 ms). Trace:
W20240506 15:14:43.220388 21032 rpcz_store.cc:269] 0506 15:14:40.983153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.168052 (+2184899us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.168150 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.168548 22331 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.165884 20996 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:43.230129 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 959) took 2169 ms. Trace:
W20240506 15:14:43.221367 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 837) took 2220 ms (client timeout 1620 ms). Trace:
W20240506 15:14:43.230221 20845 connection.cc:232] RPC call timeout handler was delayed by -1.2603878080001323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20240506 15:14:43.230386 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64587.3us
W20240506 15:14:43.219246 20921 connection.cc:232] RPC call timeout handler was delayed by -1.6374525240000821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20240506 15:14:43.231174 20996 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:42.950987 21031 rpcz_store.cc:269] 0506 15:14:39.556179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.950450 (+3394271us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.950548 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.231366 20921 connection.cc:232] RPC call timeout handler was delayed by -1.6265891480001073s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
I20240506 15:14:43.161080 20998 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:47952 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 837)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 819)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 80)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 836)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 840)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 75)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 79)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 818)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1260)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1266)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1276)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 81)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 845)
W20240506 15:14:43.161659 20950 rpcz_store.cc:269] 0506 15:14:38.228362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:38.228537 (+ 175us) service_pool.cc:225] Handling call
0506 15:14:41.732530 (+3503993us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b58000a2998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:42.947049 (+1214519us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10957568}
W20240506 15:14:43.158109 20874 rpcz_store.cc:269] 0506 15:14:41.038064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.964348 (+1926284us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.964485 (+ 137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.222770 20877 rpcz_store.cc:269] 0506 15:14:41.004668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.208742 (+1204074us) service_pool.cc:225] Handling call
0506 15:14:43.158729 (+949987us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.118299 22171 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.011s sys 0.008s
W20240506 15:14:43.159516 21029 rpcz_store.cc:269] 0506 15:14:40.975390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.158475 (+2183085us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.158602 (+ 127us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.160234 20957 rpcz_store.cc:276] 0506 15:14:41.156634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.856819 (+700185us) service_pool.cc:225] Handling call
0506 15:14:42.186010 (+329191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.235009 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1257) took 3694 ms. Trace:
W20240506 15:14:43.235031 21029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1280) took 2189 ms (client timeout 1610 ms). Trace:
W20240506 15:14:43.235769 21029 rpcz_store.cc:269] 0506 15:14:41.045902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.234869 (+2188967us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.234977 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.236759 21029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1282) took 2190 ms (client timeout 1696 ms). Trace:
W20240506 15:14:43.237118 21029 rpcz_store.cc:269] 0506 15:14:41.046468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.236603 (+2190135us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.236698 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.236771 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 80) took 2343 ms (client timeout 1810 ms). Trace:
W20240506 15:14:43.237802 21031 rpcz_store.cc:269] 0506 15:14:40.893422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.236598 (+2343176us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.236698 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.238045 20996 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:43.118562 21024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 828) took 2813 ms (client timeout 1657 ms). Trace:
W20240506 15:14:43.160337 22333 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.457s user 0.014s sys 0.033s
I20240506 15:14:43.221282 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:43.222153 20878 rpcz_store.cc:269] 0506 15:14:40.889809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.208742 (+1318933us) service_pool.cc:225] Handling call
0506 15:14:43.160075 (+951333us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.170408 21653 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.004s sys 0.022s
I20240506 15:14:43.223531 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:43.243517 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 831) took 2273 ms (client timeout 1826 ms). Trace:
W20240506 15:14:43.242837 21653 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.701s user 0.004s sys 0.022s
I20240506 15:14:43.168056 22177 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
W20240506 15:14:43.174870 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 1.45532e+06us
W20240506 15:14:43.226151 21028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 819) took 3669 ms (client timeout 3081 ms). Trace:
I20240506 15:14:43.226670 22183 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.227018 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.229099 22371 tablet_replica.cc:330] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: stopping tablet replica
I20240506 15:14:43.229036 22287 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:43.232154 20921 connection.cc:232] RPC call timeout handler was delayed by -1.3655891480000264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2304s for a response
W20240506 15:14:43.262622 20921 connection.cc:232] RPC call timeout handler was delayed by -1.3201982100000578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20240506 15:14:43.230763 21032 rpcz_store.cc:269] 0506 15:14:41.000883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.221190 (+2220307us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.221307 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.230827 20845 connection.cc:232] RPC call timeout handler was delayed by -1.2394630840000218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
I20240506 15:14:43.233057 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.237287 22331 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:43.176703 20924 leader_election.cc:334] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.687s (SENT)
W20240506 15:14:43.238662 21024 rpcz_store.cc:269] 0506 15:14:40.267190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.080014 (+2812824us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.080193 (+ 179us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.238967 20996 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:43.235404 21027 rpcz_store.cc:276] 0506 15:14:39.540755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.541049 (+ 294us) service_pool.cc:225] Handling call
0506 15:14:41.787239 (+2246190us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:41.787453 (+ 214us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:41.859635 (+ 72182us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:41.859694 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:41.860229 (+ 535us) log.cc:842] Serialized 26 byte log entry
0506 15:14:42.950025 (+1089796us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:42.950091 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:42.950162 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.229886 (+279724us) raft_consensus.cc:1671] finished
0506 15:14:43.229903 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.234950 (+ 5047us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84625,"raft.queue_time_us":72634,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":35420,"spinlock_wait_cycles":2560768,"thread_start_us":884550,"threads_started":1,"wal-append.queue_time_us":1202487}
W20240506 15:14:43.224958 20879 rpcz_store.cc:269] 0506 15:14:38.620167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.543367 (+3923200us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:42.615828 (+ 72461us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.244539 21029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 813) took 3689 ms (client timeout 3200 ms). Trace:
W20240506 15:14:43.244338 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 702442us
I20240506 15:14:43.245255 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:43.243947 21031 rpcz_store.cc:269] 0506 15:14:40.965314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.238228 (+2272914us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.238310 (+ 82us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.265331 20956 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240506 15:14:43.244127 20883 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
W20240506 15:14:43.271274 20883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 894) took 2232 ms (client timeout 2474 ms). Trace:
W20240506 15:14:43.266361 21024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 830) took 2321 ms (client timeout 2217 ms). Trace:
I20240506 15:14:43.166735 22366 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:43.246937 21028 rpcz_store.cc:269] 0506 15:14:39.557674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.225968 (+3668294us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.226090 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.240191 22171 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s user 0.013s sys 0.010s
I20240506 15:14:43.248719 20875 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
W20240506 15:14:43.284181 20875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 890) took 2292 ms (client timeout 2278 ms). Trace:
I20240506 15:14:43.250857 22177 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:43.254580 22358 log.cc:924] Time spent T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.307s user 0.000s sys 0.000s
W20240506 15:14:43.284699 20875 rpcz_store.cc:269] 0506 15:14:40.991855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.219710 (+2227855us) service_pool.cc:225] Handling call
0506 15:14:43.284094 (+ 64384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.257977 22379 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:43.258594 22368 log.cc:924] Time spent T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20240506 15:14:43.243459 22232 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:43.293426 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 956) took 2250 ms. Trace:
I20240506 15:14:43.243696 22365 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:43.241385 22335 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:43.248415 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.168370 22355 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.294561 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.308743 22271 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:43.230778 20953 rpcz_store.cc:276] 0506 15:14:41.060768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.838254 (+777486us) service_pool.cc:225] Handling call
0506 15:14:41.838636 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:41.839003 (+ 367us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:41.839033 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:41.839453 (+ 420us) log.cc:842] Serialized 26 byte log entry
0506 15:14:42.319531 (+480078us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:42.319645 (+ 114us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:42.319791 (+ 146us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.229882 (+910091us) raft_consensus.cc:1671] finished
0506 15:14:43.229906 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.230041 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":457,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":479634,"threads_started":1,"wal-append.queue_time_us":1335482}
W20240506 15:14:43.272076 21024 rpcz_store.cc:269] 0506 15:14:40.944946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.266204 (+2321258us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.266306 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.312273 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.313201 21024 raft_consensus.cc:2396] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240506 15:14:43.313705 21024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 838) took 2308 ms (client timeout 2775 ms). Trace:
W20240506 15:14:43.314168 21024 rpcz_store.cc:269] 0506 15:14:41.005110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.311964 (+2306854us) service_pool.cc:225] Handling call
0506 15:14:43.313639 (+ 1675us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.315364 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:43.266520 21030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1267) took 3387 ms (client timeout 1726 ms). Trace:
W20240506 15:14:43.267766 21027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1286) took 1593 ms (client timeout 1525 ms). Trace:
W20240506 15:14:43.316273 21030 rpcz_store.cc:269] 0506 15:14:39.879425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:40.200427 (+321002us) service_pool.cc:225] Handling call
0506 15:14:43.266326 (+3065899us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b54001ade54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:43.266446 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16259328}
W20240506 15:14:43.316706 21027 rpcz_store.cc:269] 0506 15:14:41.674471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.267589 (+1593118us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.267707 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.269114 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.318053 20949 raft_consensus.cc:2396] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240506 15:14:43.270440 21029 rpcz_store.cc:269] 0506 15:14:39.554931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.244343 (+3689412us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.244483 (+ 140us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.264578 20845 connection.cc:232] RPC call timeout handler was delayed by -1.2008212169999979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20240506 15:14:43.318981 20845 connection.cc:232] RPC call timeout handler was delayed by -1.1711915660000614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
I20240506 15:14:43.265152 20874 raft_consensus.cc:2396] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:43.173895 21970 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.265956 20924 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:43.321005 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 144309us
I20240506 15:14:43.321564 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1241) took 1138 ms. Trace:
I20240506 15:14:43.322631 20874 rpcz_store.cc:276] 0506 15:14:42.183151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.232790 (+1049639us) service_pool.cc:225] Handling call
0506 15:14:43.321502 (+ 88712us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.323058 20924 leader_election.cc:334] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.711s (SENT)
I20240506 15:14:43.323702 20924 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:43.323879 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:43.325251 20924 leader_election.cc:334] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.724s (SENT)
W20240506 15:14:43.271641 20883 rpcz_store.cc:269] 0506 15:14:41.038405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.220925 (+2182520us) service_pool.cc:225] Handling call
0506 15:14:43.271205 (+ 50280us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.267666 20921 connection.cc:232] RPC call timeout handler was delayed by -1.3197043790000862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20240506 15:14:43.320842 20949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 953) took 2280 ms (client timeout 2564 ms). Trace:
I20240506 15:14:43.271023 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1240) took 1089 ms. Trace:
W20240506 15:14:43.328150 20949 rpcz_store.cc:269] 0506 15:14:41.040776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.268836 (+2228060us) service_pool.cc:225] Handling call
0506 15:14:43.320784 (+ 51948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.328454 20878 rpcz_store.cc:276] 0506 15:14:42.181481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.242913 (+1061432us) service_pool.cc:225] Handling call
0506 15:14:43.270967 (+ 28054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.277498 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:43.265491 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 834) took 2284 ms (client timeout 1928 ms). Trace:
I20240506 15:14:43.279917 20956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 55) took 3169 ms. Trace:
W20240506 15:14:43.244722 21023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1253) took 3704 ms (client timeout 1525 ms). Trace:
W20240506 15:14:43.281487 20881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 896) took 2227 ms (client timeout 1829 ms). Trace:
I20240506 15:14:43.263830 20882 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:43.282480 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.282624 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:43.282894 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43991.9us
I20240506 15:14:43.334079 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.283435 22350 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:43.268471 22371 raft_consensus.cc:2229] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240506 15:14:43.283115 22366 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:43.266270 22331 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.304280 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1239) took 1123 ms. Trace:
W20240506 15:14:43.342808 21031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1283) took 2296 ms (client timeout 2269 ms). Trace:
W20240506 15:14:43.343262 21031 rpcz_store.cc:269] 0506 15:14:41.046747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.270999 (+2224252us) service_pool.cc:225] Handling call
0506 15:14:43.342729 (+ 71730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.343081 20877 rpcz_store.cc:276] 0506 15:14:42.181170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.232855 (+1051685us) service_pool.cc:225] Handling call
0506 15:14:43.304205 (+ 71350us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.296703 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 895) took 2258 ms (client timeout 1893 ms). Trace:
W20240506 15:14:43.344352 20880 rpcz_store.cc:269] 0506 15:14:41.039036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.296544 (+2257508us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.296636 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.293897 20954 rpcz_store.cc:276] 0506 15:14:41.042776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.109340 (+ 66564us) service_pool.cc:225] Handling call
0506 15:14:41.148389 (+ 39049us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:41.663703 (+515314us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:41.663735 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:41.669172 (+ 5437us) log.cc:842] Serialized 26 byte log entry
0506 15:14:43.161412 (+1492240us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:43.179108 (+ 17696us) log.cc:842] Serialized 14 byte log entry
0506 15:14:43.179373 (+ 265us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.179461 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.253964 (+ 74503us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:43.287831 (+ 33867us) raft_consensus.cc:1671] finished
0506 15:14:43.293210 (+ 5379us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.293365 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6602,"prepare.queue_time_us":17394,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":1997,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":41242,"spinlock_wait_cycles":240862976,"thread_start_us":1484603,"threads_started":1,"wal-append.queue_time_us":1271016}
I20240506 15:14:43.330039 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:43.345805 21030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 829) took 2451 ms (client timeout 3119 ms). Trace:
I20240506 15:14:43.346741 20878 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:43.347533 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1245) took 1163 ms. Trace:
I20240506 15:14:43.348003 20878 rpcz_store.cc:276] 0506 15:14:42.184236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.329652 (+1145416us) service_pool.cc:225] Handling call
0506 15:14:43.347452 (+ 17800us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.330792 21032 rpcz_store.cc:269] 0506 15:14:40.981462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.265309 (+2283847us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.265431 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 15:14:43.327814 20921 connection.cc:232] RPC call timeout handler was delayed by -1.3135760630000277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20240506 15:14:43.349321 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:43.349799 20921 connection.cc:232] RPC call timeout handler was delayed by -1.21258914800012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24740000000000004s for a response
I20240506 15:14:43.350548 20880 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:43.330937 20956 rpcz_store.cc:276] 0506 15:14:40.110609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.264753 (+3154144us) service_pool.cc:225] Handling call
0506 15:14:43.265116 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:43.279629 (+ 14513us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.279849 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.352346 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:43.351125 20880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 876) took 3836 ms (client timeout 4892 ms). Trace:
I20240506 15:14:43.353785 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:43.354076 20880 rpcz_store.cc:269] 0506 15:14:39.515006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.348957 (+3833951us) service_pool.cc:225] Handling call
0506 15:14:43.351053 (+ 2096us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.333024 20881 rpcz_store.cc:269] 0506 15:14:41.054117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.281145 (+2227028us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.281303 (+ 158us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.355296 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.355648 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.325289 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1244) took 1141 ms. Trace:
I20240506 15:14:43.333223 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.295794 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.334105 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.294404 22232 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:43.342190 21028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1284) took 1668 ms (client timeout 1675 ms). Trace:
I20240506 15:14:43.304140 22335 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:43.380467 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.388513 21029 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:43.389065 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 841) took 2345 ms. Trace:
I20240506 15:14:43.388993 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.389631 21029 rpcz_store.cc:276] 0506 15:14:41.043473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.318726 (+2275253us) service_pool.cc:225] Handling call
0506 15:14:43.389011 (+ 70285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.356168 20880 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:43.392640 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.394506 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.391886 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.395339 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 843) took 1185 ms. Trace:
I20240506 15:14:43.385417 20883 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.395926 20880 raft_consensus.cc:2354] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
I20240506 15:14:43.383955 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.391045 20996 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:43.304989 22365 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:43.293064 22379 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:43.346565 21030 rpcz_store.cc:269] 0506 15:14:40.894332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.333851 (+2439519us) service_pool.cc:225] Handling call
0506 15:14:43.345727 (+ 11876us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.385211 21028 rpcz_store.cc:269] 0506 15:14:41.673880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.278663 (+1604783us) service_pool.cc:225] Handling call
0506 15:14:43.342123 (+ 63460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.395222 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.321482 22271 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:43.386754 21027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 820) took 3828 ms (client timeout 4506 ms). Trace:
I20240506 15:14:43.409083 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.498410 20881 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:43.396016 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 897) took 2287 ms. Trace:
I20240506 15:14:43.483148 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.395788 21031 rpcz_store.cc:276] 0506 15:14:42.209903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.344077 (+1134174us) service_pool.cc:225] Handling call
0506 15:14:43.395270 (+ 51193us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.497983 21027 rpcz_store.cc:269] 0506 15:14:39.558239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.317334 (+3759095us) service_pool.cc:225] Handling call
0506 15:14:43.386675 (+ 69341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16128}
I20240506 15:14:43.500002 20956 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.501739 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.416100 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 966) took 2306 ms. Trace:
I20240506 15:14:43.397006 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.393507 20878 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:43.408664 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.418933 20845 service_pool.cc:138] BulkChangeConfig request on kudu.consensus.ConsensusService from 127.0.0.1:49812 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1253)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1264)
I20240506 15:14:43.345427 22187 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:43.386801 20874 rpcz_store.cc:276] 0506 15:14:42.183984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.323546 (+1139562us) service_pool.cc:225] Handling call
0506 15:14:43.325207 (+ 1661us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.433498 20876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 889) took 2444 ms (client timeout 2255 ms). Trace:
W20240506 15:14:43.554564 20876 rpcz_store.cc:269] 0506 15:14:40.989547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.433277 (+2443730us) service_pool.cc:212] Skipping call since client already timed out
0506 15:14:43.433436 (+ 159us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 15:14:43.341727 22350 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:43.420562 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:43.440477 20828 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: BulkChangeConfig request on kudu.consensus.ConsensusService from 127.0.0.1:49812 dropped due to backpressure. The service queue is full; it has 50 items.
I20240506 15:14:43.555671 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.415403 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.503527 21032 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:43.497987 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 885) took 3386 ms. Trace:
I20240506 15:14:43.556844 20877 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:43.499389 20883 rpcz_store.cc:276] 0506 15:14:41.108331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.332755 (+2224424us) service_pool.cc:225] Handling call
0506 15:14:43.395944 (+ 63189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.503087 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.342890 22046 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:43.345180 20882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 887) took 2354 ms (client timeout 2800 ms). Trace:
I20240506 15:14:43.517015 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:43.540319 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 149377us
I20240506 15:14:43.556314 20874 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:43.427271 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.562157 20955 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.563216 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.543258 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.559394 21690 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:43.443773 22177 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:43.567346 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.544813 20875 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:43.388104 22335 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:43.390714 22298 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.155s
I20240506 15:14:43.578898 22335 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.547406 22355 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:43.548439 22383 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20240506 15:14:43.473913 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:43.549116 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 222662us
W20240506 15:14:43.602782 20924 leader_election.cc:334] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.736s (SENT)
I20240506 15:14:43.603209 20924 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:43.551132 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.551533 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.604139 20952 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:43.552642 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.605710 20878 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:43.606978 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.427021 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.502786 20958 rpcz_store.cc:276] 0506 15:14:41.109699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:41.844606 (+734907us) service_pool.cc:225] Handling call
0506 15:14:41.844922 (+ 316us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:41.848471 (+ 3549us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:41.848500 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:41.848901 (+ 401us) log.cc:842] Serialized 26 byte log entry
0506 15:14:42.303489 (+454588us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:42.303551 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:42.303649 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.415821 (+1112172us) raft_consensus.cc:1671] finished
0506 15:14:43.415867 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.416038 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3008,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":454130,"threads_started":1,"wal-append.queue_time_us":1200525}
I20240506 15:14:43.556892 20880 rpcz_store.cc:276] 0506 15:14:40.111159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.391488 (+3280329us) service_pool.cc:225] Handling call
0506 15:14:43.497927 (+106439us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.609892 20924 leader_election.cc:334] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.739s (SENT)
I20240506 15:14:43.557929 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.610551 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:43.527084 20954 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.610453 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:43.558051 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.390436 22232 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:43.445135 22376 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240506 15:14:43.445914 22385 log.cc:924] Time spent T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20240506 15:14:43.557979 20949 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.618523 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:43.541520 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.619002 21031 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:43.619256 21028 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:43.544201 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.620571 21027 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:43.621254 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 836) took 2630 ms. Trace:
I20240506 15:14:43.621829 21027 rpcz_store.cc:276] 0506 15:14:40.990828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.501343 (+2510515us) service_pool.cc:225] Handling call
0506 15:14:43.621180 (+119837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.564767 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.623181 20953 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:43.561944 20845 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1253)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 901)
I20240506 15:14:43.458144 22371 raft_consensus.cc:2258] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240506 15:14:43.567498 20996 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:43.568300 21032 raft_consensus.cc:2396] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:43.380690 22071 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:43.629321 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.629832 22335 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:43.514640 22231 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:43.630513 21027 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
W20240506 15:14:43.475016 22364 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.527s user 0.012s sys 0.000s
I20240506 15:14:43.569749 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.440843 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.572482 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1243) took 1388 ms. Trace:
W20240506 15:14:43.561192 20882 rpcz_store.cc:269] 0506 15:14:40.987875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.226733 (+2238858us) service_pool.cc:225] Handling call
0506 15:14:43.341983 (+115250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:43.336400 21023 rpcz_store.cc:269] 0506 15:14:39.537154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.543262 (+ 6108us) service_pool.cc:225] Handling call
0506 15:14:41.794930 (+2251668us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:43.240154 (+1445224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97005440}
I20240506 15:14:43.578749 21970 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.579444 22287 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.434041 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:43.476290 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.545318 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.650213 21030 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:43.651199 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 839) took 2640 ms. Trace:
I20240506 15:14:43.580515 20881 raft_consensus.cc:2396] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240506 15:14:43.580439 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:43.510958 22365 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:43.474391 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.650048 22287 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.583261 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:43.555290 22350 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:43.579813 22186 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:43.585731 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.585805 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1256) took 4045 ms. Trace:
I20240506 15:14:43.586485 22375 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.665467 22375 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.584856 22183 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:43.667709 22375 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.587893 22366 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.564615 22046 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.588972 22230 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.668434 22375 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:43.668936 22375 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.591346 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.592531 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:43.593695 22187 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240506 15:14:43.598462 22384 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.297s user 0.004s sys 0.000s
I20240506 15:14:43.563668 22338 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20240506 15:14:43.631784 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64327us
I20240506 15:14:43.655660 22331 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.632105 20950 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:43.680030 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.617149 22361 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:43.680814 21023 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:43.617336 22355 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:43.637979 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.686635 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1285) took 2012 ms. Trace:
I20240506 15:14:43.686986 21028 rpcz_store.cc:276] 0506 15:14:41.674198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.620183 (+1945985us) service_pool.cc:225] Handling call
0506 15:14:43.686570 (+ 66387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.688501 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 75) took 4142 ms. Trace:
I20240506 15:14:43.688947 21028 rpcz_store.cc:276] 0506 15:14:39.545952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.687627 (+4141675us) service_pool.cc:225] Handling call
0506 15:14:43.687969 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.688131 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.688164 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:43.688201 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:43.688234 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.688322 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.688442 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.644462 21031 raft_consensus.cc:2396] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20240506 15:14:43.627799 22310 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:43.651655 21030 rpcz_store.cc:276] 0506 15:14:41.011150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.499680 (+2488530us) service_pool.cc:225] Handling call
0506 15:14:43.651082 (+151402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.633040 22260 raft_consensus.cc:2684] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Received pre-election callback for term 5 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20240506 15:14:43.634690 22341 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:43.668030 22183 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.667685 22350 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.662118 21025 rpcz_store.cc:276] 0506 15:14:39.540349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.543067 (+ 2718us) service_pool.cc:225] Handling call
0506 15:14:42.961065 (+3417998us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:42.961320 (+ 255us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:42.961812 (+ 492us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:42.961848 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:42.962471 (+ 623us) log.cc:842] Serialized 26 byte log entry
0506 15:14:43.177137 (+214666us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:43.219571 (+ 42434us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.240480 (+ 20909us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.555991 (+315511us) raft_consensus.cc:1671] finished
0506 15:14:43.560172 (+ 4181us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.585717 (+ 25545us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42404,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":38346,"spinlock_wait_cycles":97126144,"thread_start_us":205276,"threads_started":1,"wal-append.queue_time_us":330258}
I20240506 15:14:43.668936 22230 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.710417 21026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1254) took 4173 ms. Trace:
I20240506 15:14:43.711767 22230 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.582896 22171 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:43.690129 22391 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:43.712517 22171 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.712680 22391 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.714660 22171 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.714921 22391 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.710949 21026 rpcz_store.cc:276] 0506 15:14:39.537604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:39.543290 (+ 5686us) service_pool.cc:225] Handling call
0506 15:14:41.698442 (+2155152us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b58000a2c98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:41.698654 (+ 212us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:42.179060 (+480406us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:42.207011 (+ 27951us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:42.491984 (+284973us) log.cc:842] Serialized 26 byte log entry
0506 15:14:43.168422 (+676438us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:43.176129 (+ 7707us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.190782 (+ 14653us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:43.673418 (+482636us) raft_consensus.cc:1671] finished
0506 15:14:43.675472 (+ 2054us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.710348 (+ 34876us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35484,"raft.queue_time_us":35008,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":124988,"spinlock_wait_cycles":97319680,"thread_start_us":815949,"threads_started":2,"wal-append.queue_time_us":678970}
I20240506 15:14:43.629348 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 840) took 2589 ms. Trace:
I20240506 15:14:43.633056 21653 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:43.718108 22183 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.718770 22371 ts_tablet_manager.cc:1875] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:14:43.719251 22366 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:43.719453 21025 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 3. Ops: []
I20240506 15:14:43.719727 22366 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:43.720213 22366 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:43.721025 22375 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.718323 21032 rpcz_store.cc:276] 0506 15:14:41.040429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.556907 (+2516478us) service_pool.cc:225] Handling call
0506 15:14:43.629274 (+ 72367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.670020 22046 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.709261 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 78) took 4162 ms. Trace:
I20240506 15:14:43.716324 22287 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.690223 22331 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:43.722537 21032 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:43.721856 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 810) took 4172 ms. Trace:
I20240506 15:14:43.723184 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 818) took 4166 ms. Trace:
I20240506 15:14:43.723441 21025 rpcz_store.cc:276] 0506 15:14:39.549202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.718916 (+4169714us) service_pool.cc:225] Handling call
0506 15:14:43.719233 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.721587 (+ 2354us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.721788 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.671558 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:43.705574 22310 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:43.680747 22379 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:43.709216 22341 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:43.724751 22310 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:43.723593 21032 rpcz_store.cc:276] 0506 15:14:39.556546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.722062 (+4165516us) service_pool.cc:225] Handling call
0506 15:14:43.722357 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.722963 (+ 606us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.723126 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.725133 22341 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:43.725942 22331 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:43.718592 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 85619.9us
I20240506 15:14:43.727223 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 76) took 4180 ms. Trace:
I20240506 15:14:43.717978 21026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 79) took 4170 ms. Trace:
I20240506 15:14:43.717939 22350 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:43.701395 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 842) took 2644 ms. Trace:
I20240506 15:14:43.722566 21030 rpcz_store.cc:276] 0506 15:14:39.547263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.705118 (+4157855us) service_pool.cc:225] Handling call
0506 15:14:43.708774 (+ 3656us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.708954 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.708986 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:43.709029 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:43.709059 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.709116 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.709219 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.730006 21031 rpcz_store.cc:276] 0506 15:14:41.056884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.604787 (+2547903us) service_pool.cc:225] Handling call
0506 15:14:43.701313 (+ 96526us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.731287 21030 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:43.731884 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1265) took 3853 ms. Trace:
I20240506 15:14:43.732769 21030 rpcz_store.cc:276] 0506 15:14:39.878842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.730397 (+3851555us) service_pool.cc:225] Handling call
0506 15:14:43.730973 (+ 576us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.731637 (+ 664us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.731830 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.726027 21032 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:43.716007 22365 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.735262 22341 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.738257 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1260) took 4170 ms. Trace:
I20240506 15:14:43.739130 21032 rpcz_store.cc:276] 0506 15:14:39.569711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.725405 (+4155694us) service_pool.cc:225] Handling call
0506 15:14:43.725699 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.737952 (+ 12253us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.738202 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.809391 21029 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:43.737982 21031 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:43.633913 20845 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:50460 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1253)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 901)
I20240506 15:14:43.633318 20877 raft_consensus.cc:2362] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2. Re-sending same reply.
I20240506 15:14:43.636648 20879 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:43.610589 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.653098 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.613371 20883 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:43.656132 20874 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:43.634254 20875 rpcz_store.cc:276] 0506 15:14:42.183732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.287541 (+1103809us) service_pool.cc:225] Handling call
0506 15:14:43.572410 (+284869us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.613543 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:43.616413 20921 leader_election.cc:334] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Remote error: Timed out: Call waited in the queue past client deadline
W20240506 15:14:43.612565 20924 connection.cc:232] RPC call timeout handler was delayed by -1.605230269000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
I20240506 15:14:43.789979 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:43.610848 20876 raft_consensus.cc:2362] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:43.677685 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:43.727860 21028 rpcz_store.cc:276] 0506 15:14:39.546545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.689862 (+4143317us) service_pool.cc:225] Handling call
0506 15:14:43.726728 (+ 36866us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.726935 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.726962 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:43.727001 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:43.727027 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.727076 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.727172 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.729419 21026 rpcz_store.cc:276] 0506 15:14:39.547582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.717084 (+4169502us) service_pool.cc:225] Handling call
0506 15:14:43.717466 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.717656 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.717688 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:43.717735 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:43.717773 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.717830 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.717926 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:43.811190 21032 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:14:44.012966 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 335152us
I20240506 15:14:44.015128 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:44.017572 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.019767 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.022082 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.023186 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:44.024726 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.025753 20880 raft_consensus.cc:2362] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2. Re-sending same reply.
I20240506 15:14:44.026948 20878 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:43.812227 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1266) took 3932 ms. Trace:
I20240506 15:14:44.028432 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.029553 20881 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:44.030836 20879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1242) took 1847 ms (client timeout 1856 ms). Trace:
I20240506 15:14:44.031710 20882 raft_consensus.cc:2362] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3. Re-sending same reply.
I20240506 15:14:43.975850 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 824) took 4041 ms. Trace:
I20240506 15:14:44.035466 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 807) took 4487 ms. Trace:
I20240506 15:14:43.735103 22232 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:43.742630 22251 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:44.053225 22251 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:44.054726 20924 connection.cc:232] RPC call timeout handler was delayed by -1.4565204410000661s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20240506 15:14:44.055395 20924 connection.cc:232] RPC call timeout handler was delayed by -1.4549302690001014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20240506 15:14:44.055651 21024 raft_consensus.cc:2354] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
W20240506 15:14:44.055930 20924 connection.cc:232] RPC call timeout handler was delayed by -1.3935977310001135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20240506 15:14:44.056427 20924 connection.cc:232] RPC call timeout handler was delayed by -1.3656204410001465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20240506 15:14:44.056803 20924 connection.cc:232] RPC call timeout handler was delayed by -1.3442302690000361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2304s for a response
I20240506 15:14:43.740193 22341 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:44.057169 20924 connection.cc:232] RPC call timeout handler was delayed by -1.3080204410000533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18230000000000002s for a response
I20240506 15:14:43.739848 22365 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.057374 21687 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:44.057672 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2900204410001379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
I20240506 15:14:43.740723 22366 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:44.058254 20924 connection.cc:232] RPC call timeout handler was delayed by -1.278320441000119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
W20240506 15:14:44.058715 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2522204410000768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20240506 15:14:44.059417 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2441204410000637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20240506 15:14:44.059851 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2378204410000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20240506 15:14:44.060220 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1259) took 4515 ms. Trace:
W20240506 15:14:44.001047 20767 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 221742us
I20240506 15:14:44.039976 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1271) took 4101 ms. Trace:
I20240506 15:14:44.043015 21031 rpcz_store.cc:276] 0506 15:14:39.879138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.731039 (+3851901us) service_pool.cc:225] Handling call
0506 15:14:43.731296 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.811227 (+ 79931us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.811969 (+ 742us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.045236 20874 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
W20240506 15:14:44.061369 20924 connection.cc:232] RPC call timeout handler was delayed by -1.221697731000063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20240506 15:14:44.062122 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2171204410001337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
I20240506 15:14:44.046157 21029 rpcz_store.cc:276] 0506 15:14:39.547909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.717382 (+4169473us) service_pool.cc:225] Handling call
0506 15:14:43.717664 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.810344 (+ 92680us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.003622 (+193278us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b54000c8d54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:44.035336 (+ 31714us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208883456}
I20240506 15:14:44.061506 21032 rpcz_store.cc:276] 0506 15:14:39.938927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.809505 (+3870578us) service_pool.cc:225] Handling call
0506 15:14:43.810236 (+ 731us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.039689 (+229453us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.039923 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.065254 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2162204410001323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
W20240506 15:14:44.067798 20924 connection.cc:232] RPC call timeout handler was delayed by -1.1946204410000973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20240506 15:14:44.068495 22227 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:44.069887 20883 raft_consensus.cc:2354] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:43.753440 22310 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.075390 20924 connection.cc:232] RPC call timeout handler was delayed by -1.1912302690001297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24740000000000004s for a response
I20240506 15:14:44.068692 21686 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240506 15:14:44.081075 20877 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:44.077070 20924 connection.cc:232] RPC call timeout handler was delayed by -1.0749204410001312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20240506 15:14:44.071883 20876 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:44.077412 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.079702 22071 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:44.079408 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.069897 22227 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:44.045984 21030 rpcz_store.cc:276] 0506 15:14:39.934114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.736486 (+3802372us) service_pool.cc:225] Handling call
0506 15:14:43.736891 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.737153 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:43.937251 (+200098us) log.cc:842] Serialized 14 byte log entry
0506 15:14:43.974989 (+ 37738us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:43.975090 (+ 101us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:43.975193 (+ 103us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.975322 (+ 129us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:43.975591 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":276312,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":79,"raft.queue_time_us":74619,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":368,"thread_start_us":236361,"threads_started":2,"wal-append.queue_time_us":65715}
I20240506 15:14:44.086730 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:44.044625 20879 rpcz_store.cc:269] 0506 15:14:42.183456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.269242 (+1085786us) service_pool.cc:225] Handling call
0506 15:14:43.473727 (+204485us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b58002c4998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:44.030565 (+556838us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4507904}
I20240506 15:14:44.086138 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.083348 22366 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.088215 22227 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.079061 20875 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.091202 20878 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.089177 22264 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:44.106009 22354 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240506 15:14:44.106817 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 91884.6us
I20240506 15:14:44.106595 22354 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:44.108894 22354 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:44.106683 20845 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 4 similar messages]
I20240506 15:14:44.108304 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:44.112941 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 496456us
I20240506 15:14:44.113566 20880 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.119202 20874 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.060705 21023 rpcz_store.cc:276] 0506 15:14:39.548895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.681976 (+4133081us) service_pool.cc:225] Handling call
0506 15:14:43.682578 (+ 602us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:43.683007 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.683039 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:43.683559 (+ 520us) log.cc:842] Serialized 26 byte log entry
0506 15:14:43.807951 (+124392us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:43.808122 (+ 171us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:43.808420 (+ 298us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:44.059669 (+251249us) raft_consensus.cc:1671] finished
0506 15:14:44.059700 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.060059 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7460,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":2195,"thread_start_us":123373,"threads_started":1,"wal-append.queue_time_us":373050}
I20240506 15:14:44.097529 22071 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.128466 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.129509 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1276) took 4118 ms. Trace:
I20240506 15:14:44.130261 22403 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:44.130019 21030 rpcz_store.cc:276] 0506 15:14:40.011164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.107437 (+4096273us) service_pool.cc:225] Handling call
0506 15:14:44.125772 (+ 18335us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.128926 (+ 3154us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.129048 (+ 122us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.129157 (+ 109us) raft_consensus.cc:1631] Marking committed up to 6
0506 15:14:44.129209 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.129287 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.129418 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.122989 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.134634 22403 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.135321 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.136080 20874 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:44.137241 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.126406 20877 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.137712 22403 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.130031 20879 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.132011 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.138063 20874 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.130743 20921 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:44.143703 20874 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240506 15:14:44.138139 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.140551 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.149268 20878 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.141175 20883 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:44.153682 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 873) took 5614 ms. Trace:
I20240506 15:14:44.154434 21023 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:44.150175 20880 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.155323 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 844) took 1945 ms. Trace:
I20240506 15:14:44.158424 21023 rpcz_store.cc:276] 0506 15:14:42.210296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.143922 (+1933626us) service_pool.cc:225] Handling call
0506 15:14:44.154107 (+ 10185us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.155027 (+ 920us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.155234 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.160950 21023 raft_consensus.cc:1226] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:44.161855 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 845) took 1901 ms. Trace:
I20240506 15:14:44.162348 21023 rpcz_store.cc:276] 0506 15:14:42.260206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.159795 (+1899589us) service_pool.cc:225] Handling call
0506 15:14:44.160523 (+ 728us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.161464 (+ 941us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.161750 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.156603 20846 connection.cc:232] RPC call timeout handler was delayed by -2.8664232489999222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20240506 15:14:44.157016 20882 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.134361 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.165880 20875 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.145494 20845 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 919)
Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50460 (request call id 323)
W20240506 15:14:44.166358 20846 connection.cc:232] RPC call timeout handler was delayed by -2.7971232489999238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20240506 15:14:44.167488 20846 connection.cc:232] RPC call timeout handler was delayed by -2.6846232489999693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1789s for a response
I20240506 15:14:44.167843 20875 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240506 15:14:44.168071 20846 connection.cc:232] RPC call timeout handler was delayed by -2.6342232490000015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
I20240506 15:14:44.145439 22403 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:44.164819 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1264) took 4286 ms. Trace:
I20240506 15:14:44.170281 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.154337 20883 rpcz_store.cc:276] 0506 15:14:38.539057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.151617 (+5612560us) service_pool.cc:225] Handling call
0506 15:14:44.152211 (+ 594us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.152688 (+ 477us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.152730 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.152825 (+ 95us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:44.152894 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.153024 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.153557 (+ 533us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.174618 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1187) took 5655 ms. Trace:
I20240506 15:14:44.156370 21687 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
W20240506 15:14:44.176265 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 45713.6us
I20240506 15:14:44.178576 20874 rpcz_store.cc:276] 0506 15:14:38.516401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.142761 (+5626360us) service_pool.cc:225] Handling call
0506 15:14:44.143330 (+ 569us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:44.171232 (+ 27902us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.171536 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.176607 21027 rpcz_store.cc:276] 0506 15:14:39.878555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.729325 (+3850770us) service_pool.cc:225] Handling call
0506 15:14:43.729611 (+ 286us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:43.729955 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:43.729984 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:43.730578 (+ 594us) log.cc:842] Serialized 26 byte log entry
0506 15:14:44.123712 (+393134us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:44.132320 (+ 8608us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.134655 (+ 2335us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:44.164337 (+ 29682us) raft_consensus.cc:1671] finished
0506 15:14:44.164360 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.164645 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1205,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":1740,"thread_start_us":347786,"threads_started":1,"wal-append.queue_time_us":416359}
I20240506 15:14:44.176941 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 874) took 5596 ms. Trace:
I20240506 15:14:44.184204 20880 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.189812 20878 rpcz_store.cc:276] 0506 15:14:38.578075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.157087 (+5579012us) service_pool.cc:225] Handling call
0506 15:14:44.157606 (+ 519us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.157967 (+ 361us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.171555 (+ 13588us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.171624 (+ 69us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:44.173672 (+ 2048us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.173757 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.173898 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.187636 20846 connection.cc:232] RPC call timeout handler was delayed by -2.566619196999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29760000000000003s for a response
I20240506 15:14:44.180065 22403 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.202045 22071 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:44.204350 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1191) took 5483 ms. Trace:
I20240506 15:14:44.205067 20875 rpcz_store.cc:276] 0506 15:14:38.710804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.167003 (+5456199us) service_pool.cc:225] Handling call
0506 15:14:44.167550 (+ 547us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:44.178401 (+ 10851us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.194089 (+ 15688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.210942 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:44.211228 22365 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:44.211498 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:44.212443 20846 connection.cc:232] RPC call timeout handler was delayed by -2.5086095719999548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
I20240506 15:14:44.212213 22365 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.212594 22338 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:44.212877 20846 connection.cc:232] RPC call timeout handler was delayed by -2.4695232489999626s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
W20240506 15:14:44.213330 20846 connection.cc:232] RPC call timeout handler was delayed by -2.4462286770000219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
W20240506 15:14:44.213685 20846 connection.cc:232] RPC call timeout handler was delayed by -2.2247232489999078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22999999999999998s for a response
W20240506 15:14:44.214174 20846 connection.cc:232] RPC call timeout handler was delayed by -1.9907232489999842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.256s for a response
I20240506 15:14:44.213518 22338 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.214537 20846 connection.cc:232] RPC call timeout handler was delayed by -1.9259232489999931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26320000000000005s for a response
W20240506 15:14:44.214866 20846 connection.cc:232] RPC call timeout handler was delayed by -1.4147232489999624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32000000000000006s for a response
W20240506 15:14:44.215234 20846 connection.cc:232] RPC call timeout handler was delayed by -1.2891095719999157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30810000000000004s for a response
I20240506 15:14:44.214761 22365 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.226122 22371 ts_tablet_manager.cc:1888] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240506 15:14:44.226900 22371 log.cc:1196] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-2-root/wals/93fe2d6a721f4eaeb4b3f4b5f71e1a91
I20240506 15:14:44.227361 22232 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.237233 22271 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:44.237918 22271 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:44.238373 22271 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:44.170292 22231 consensus_queue.cc:1046] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20240506 15:14:44.192806 22366 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20240506 15:14:44.242954 22071 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:44.243512 21030 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:14:44.243682 22187 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:44.244352 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 81) took 2052 ms. Trace:
I20240506 15:14:44.244769 21030 rpcz_store.cc:276] 0506 15:14:42.191877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.145695 (+1953818us) service_pool.cc:225] Handling call
0506 15:14:44.243061 (+ 97366us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.244050 (+ 989us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.244297 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.244376 22393 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:44.246245 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.247838 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:44.248857 20877 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:44.199853 22403 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.258072 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 871) took 5740 ms. Trace:
I20240506 15:14:44.260324 20877 rpcz_store.cc:276] 0506 15:14:38.512530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.146853 (+5634323us) service_pool.cc:225] Handling call
0506 15:14:44.248451 (+101598us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.252707 (+ 4256us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.252950 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.264475 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.238459 22338 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.275681 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.287514 22251 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.290225 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.262002 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.274246 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1273) took 4278 ms. Trace:
W20240506 15:14:44.289912 21024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1287) took 2615 ms (client timeout 2151 ms). Trace:
I20240506 15:14:44.302445 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.303125 20880 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240506 15:14:44.298614 21024 rpcz_store.cc:269] 0506 15:14:41.674760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.315034 (+1640274us) service_pool.cc:225] Handling call
0506 15:14:44.289611 (+974577us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b5400139a54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:44.289794 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1699691904}
I20240506 15:14:44.302614 21030 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:44.303381 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.307646 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1194) took 5169 ms. Trace:
I20240506 15:14:44.308900 20880 rpcz_store.cc:276] 0506 15:14:39.138307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.260902 (+5122595us) service_pool.cc:225] Handling call
0506 15:14:44.302902 (+ 42000us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:44.307135 (+ 4233us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.307545 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.244771 22295 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20240506 15:14:44.300506 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:44.318006 20881 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:14:44.319485 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.320729 20881 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.320412 22412 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20240506 15:14:44.321161 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 80486.2us
I20240506 15:14:44.323001 20921 leader_election.cc:398] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:44.323437 20921 leader_election.cc:401] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:44.325295 20878 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.319729 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1291) took 1087 ms. Trace:
I20240506 15:14:44.327713 20883 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.300032 21032 rpcz_store.cc:276] 0506 15:14:39.996329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.078890 (+4082561us) service_pool.cc:225] Handling call
0506 15:14:44.110728 (+ 31838us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.155847 (+ 45119us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.156236 (+ 389us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.273801 (+117565us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.273874 (+ 73us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.273929 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.274015 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.274171 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":233,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"raft.queue_time_us":11614,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":9111,"thread_start_us":106182,"threads_started":1}
I20240506 15:14:44.330031 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:44.332676 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.336284 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.337939 20880 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.340339 20876 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.345060 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:44.350816 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 88124.1us
I20240506 15:14:44.355835 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.353602 20877 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.328173 20882 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.360754 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.337579 22371 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47936 (request call id 454) took 2144 ms. Trace:
I20240506 15:14:44.365679 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1275) took 4363 ms. Trace:
I20240506 15:14:44.366855 20878 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.383905 20881 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.355322 22338 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:44.393414 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1274) took 4393 ms. Trace:
I20240506 15:14:44.383212 21031 rpcz_store.cc:276] 0506 15:14:40.001961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.081806 (+4079845us) service_pool.cc:225] Handling call
0506 15:14:44.112825 (+ 31019us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.152684 (+ 39859us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.153159 (+ 475us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.365411 (+212252us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.365465 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.365496 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.365544 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.365638 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":196,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":50,"raft.queue_time_us":5469,"thread_start_us":206420,"threads_started":1}
I20240506 15:14:44.357947 22264 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.366199 22403 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:44.406885 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:44.411984 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 45133.7us
I20240506 15:14:44.393950 21029 rpcz_store.cc:276] 0506 15:14:39.999673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.080138 (+4080465us) service_pool.cc:225] Handling call
0506 15:14:44.091532 (+ 11394us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.170733 (+ 79201us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.186786 (+ 16053us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.392984 (+206198us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.393048 (+ 64us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.393102 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.393185 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.393345 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4705,"prepare.queue_time_us":18835,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":113725,"threads_started":1}
I20240506 15:14:44.387279 22338 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:44.416968 22338 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:44.328701 21023 rpcz_store.cc:276] 0506 15:14:43.233266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.164048 (+930782us) service_pool.cc:225] Handling call
0506 15:14:44.168671 (+ 4623us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.176534 (+ 7863us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.232781 (+ 56247us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.319345 (+ 86564us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.319418 (+ 73us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.319460 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.319540 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.319672 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8941,"prepare.queue_time_us":58347,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":127,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":85917,"threads_started":1,"wal-append.queue_time_us":85313}
I20240506 15:14:44.421154 22271 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.421447 22227 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.426771 22227 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.428377 22271 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.434201 22310 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.436970 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 822) took 4548 ms. Trace:
I20240506 15:14:44.437476 21025 rpcz_store.cc:276] 0506 15:14:39.806852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:43.727412 (+3920560us) service_pool.cc:225] Handling call
0506 15:14:43.727694 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:43.727898 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.075301 (+347403us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.255955 (+180654us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.256039 (+ 84us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.275951 (+ 19912us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.328512 (+ 52561us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.354656 (+ 26144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4031,"prepare.queue_time_us":284964,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":193,"raft.queue_time_us":320,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":6726,"thread_start_us":472091,"threads_started":2,"wal-append.queue_time_us":202142}
I20240506 15:14:44.440621 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.441505 20876 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.438618 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1292) took 1205 ms. Trace:
I20240506 15:14:44.448829 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:44.449419 22405 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20240506 15:14:44.449625 20877 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:44.452591 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 185049us
I20240506 15:14:44.458753 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.459686 20881 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.460652 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.461290 20881 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.462085 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.413960 20880 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
W20240506 15:14:44.464690 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 54299.1us
I20240506 15:14:44.443307 21027 rpcz_store.cc:276] 0506 15:14:43.233570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.220083 (+986513us) service_pool.cc:225] Handling call
0506 15:14:44.220611 (+ 528us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.221053 (+ 442us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:14:44.270642 (+ 49589us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.438149 (+167507us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.438222 (+ 73us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:44.438277 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.438369 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.438538 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68819,"prepare.queue_time_us":47311,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":15246,"raft.queue_time_us":38645,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":140,"thread_start_us":89166,"threads_started":1}
W20240506 15:14:44.484440 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 183322us
I20240506 15:14:44.413273 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.428179 22366 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20240506 15:14:44.485934 21030 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:44.486483 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.499567 20883 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:44.489287 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.364816 22371 rpcz_store.cc:276] 0506 15:14:42.181505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:42.181689 (+ 184us) service_pool.cc:225] Handling call
0506 15:14:43.189968 (+1008279us) ts_tablet_manager.cc:1120] Deleting tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91
0506 15:14:43.228680 (+ 38712us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 15:14:44.052177 (+823497us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:44.225677 (+173500us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:44.317507 (+ 91830us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1365,"tablet-delete.queue_time_us":979901,"thread_start_us":1036534,"threads_started":1}
I20240506 15:14:44.502746 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:44.502125 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 81167.1us
I20240506 15:14:44.505360 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.509640 20874 raft_consensus.cc:2362] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3. Re-sending same reply.
I20240506 15:14:44.512544 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.518361 20881 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:44.522636 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 67) took 7991 ms. Trace:
I20240506 15:14:44.523021 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.523195 21653 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:44.524672 22251 raft_consensus.cc:3032] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:44.525138 22251 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:44.525166 22231 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:44.523265 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.527175 20874 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:44.515605 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:44.536543 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.541783 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:44.541973 20878 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:44.547019 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:44.556612 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 78492.3us
I20240506 15:14:44.557924 20883 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.558768 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1198) took 5414 ms. Trace:
I20240506 15:14:44.563083 22331 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:44.562474 20875 rpcz_store.cc:276] 0506 15:14:39.139568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.206387 (+5066819us) service_pool.cc:225] Handling call
0506 15:14:44.207052 (+ 665us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.207486 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:14:44.398991 (+191505us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.535434 (+136443us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.538943 (+ 3509us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:44.540743 (+ 1800us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.543752 (+ 3009us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.548815 (+ 5063us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":232938,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":170,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":1532,"thread_start_us":308357,"threads_started":2}
I20240506 15:14:44.561939 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:44.566298 20881 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.581410 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.562799 20922 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:44.584435 20880 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
W20240506 15:14:44.585796 20924 leader_election.cc:334] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.759s (SENT)
I20240506 15:14:44.586447 20874 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20240506 15:14:44.588399 22271 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:44.588835 22271 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:44.588914 22415 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:44.589409 22415 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:44.589855 22271 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.590245 22415 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.591488 22271 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:44.591652 22415 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:44.597136 20878 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.598071 20764 catalog_manager.cc:4830] TS 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): tablet 93fe2d6a721f4eaeb4b3f4b5f71e1a91 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
W20240506 15:14:44.598762 20764 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 65055.6us
I20240506 15:14:44.598804 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.600731 20883 raft_consensus.cc:1226] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:44.562572 20882 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.603611 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 64) took 5441 ms. Trace:
W20240506 15:14:44.606405 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 74299.6us
I20240506 15:14:44.609037 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1204) took 5450 ms. Trace:
I20240506 15:14:44.608762 20874 rpcz_store.cc:276] 0506 15:14:39.154093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.547115 (+5393022us) service_pool.cc:225] Handling call
0506 15:14:44.573144 (+ 26029us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:44.595657 (+ 22513us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.595883 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.609719 20883 rpcz_store.cc:276] 0506 15:14:39.158216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.559608 (+5401392us) service_pool.cc:225] Handling call
0506 15:14:44.584304 (+ 24696us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.608718 (+ 24414us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.608972 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.533282 22251 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.604212 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1203) took 5439 ms. Trace:
I20240506 15:14:44.611125 20878 rpcz_store.cc:276] 0506 15:14:39.157967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.544244 (+5386277us) service_pool.cc:225] Handling call
0506 15:14:44.584032 (+ 39788us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.597562 (+ 13530us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.597736 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.624950 20880 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.625792 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1205) took 5467 ms. Trace:
I20240506 15:14:44.626147 20880 rpcz_store.cc:276] 0506 15:14:39.158480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.585007 (+5426527us) service_pool.cc:225] Handling call
0506 15:14:44.623014 (+ 38007us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.625464 (+ 2450us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.625735 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.627256 22409 log.cc:924] Time spent T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.110s user 0.003s sys 0.000s
I20240506 15:14:44.629606 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.630347 20846 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:44.523213 20779 rpcz_store.cc:276] 0506 15:14:36.465200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:36.497854 (+ 32654us) service_pool.cc:225] Handling call
0506 15:14:44.456150 (+7958296us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:43.246159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:43.282899 (+ 36740us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:43.282946 (+ 47us) write_op.cc:435] Acquired schema lock
0506 15:14:43.282965 (+ 19us) tablet.cc:660] Decoding operations
0506 15:14:43.283324 (+ 359us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:43.283379 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:43.283408 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:14:43.283600 (+ 192us) tablet.cc:699] Row locks acquired
0506 15:14:43.283623 (+ 23us) write_op.cc:260] PREPARE: finished
0506 15:14:43.283767 (+ 144us) write_op.cc:270] Start()
0506 15:14:43.283868 (+ 101us) write_op.cc:276] Timestamp: P: 1715008483283717 usec, L: 0
0506 15:14:43.283883 (+ 15us) op_driver.cc:331] REPLICATION: starting
0506 15:14:43.284524 (+ 641us) log.cc:842] Serialized 535 byte log entry
0506 15:14:43.657243 (+372719us) op_driver.cc:444] REPLICATION: finished
0506 15:14:44.079528 (+422285us) write_op.cc:301] APPLY: starting
0506 15:14:44.079757 (+ 229us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:44.079987 (+ 230us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:44.092151 (+ 12164us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:44.102355 (+ 10204us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:44.102811 (+ 456us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:14:44.108576 (+ 5765us) write_op.cc:312] APPLY: finished
0506 15:14:44.109035 (+ 459us) log.cc:842] Serialized 22 byte log entry
0506 15:14:44.110471 (+ 1436us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:44.111688 (+ 1217us) write_op.cc:454] Released schema lock
0506 15:14:44.111780 (+ 92us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:44.130828 (+ 19048us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:44.130860 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3790959,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":400811,"apply.run_cpu_time_us":4565,"apply.run_wall_time_us":67919,"num_ops":1,"prepare.queue_time_us":298881,"prepare.run_cpu_time_us":32798,"prepare.run_wall_time_us":295383,"raft.queue_time_us":57704,"raft.run_cpu_time_us":8067,"raft.run_wall_time_us":115658,"replication_time_us":366396,"thread_start_us":524022,"threads_started":4,"wal-append.queue_time_us":205094}]]}
I20240506 15:14:44.635399 20878 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.635721 21027 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:44.636148 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1210) took 5476 ms. Trace:
I20240506 15:14:44.636600 20878 rpcz_store.cc:276] 0506 15:14:39.159786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.622606 (+5462820us) service_pool.cc:225] Handling call
0506 15:14:44.633718 (+ 11112us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.635874 (+ 2156us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.636093 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.637388 20875 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.638053 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1207) took 5479 ms. Trace:
I20240506 15:14:44.638092 20878 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.638199 22338 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.638603 20875 rpcz_store.cc:276] 0506 15:14:39.158993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.600333 (+5441340us) service_pool.cc:225] Handling call
0506 15:14:44.637018 (+ 36685us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.637790 (+ 772us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.637985 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.639147 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1213) took 5466 ms. Trace:
I20240506 15:14:44.639571 20878 rpcz_store.cc:276] 0506 15:14:39.172732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.637397 (+5464665us) service_pool.cc:225] Handling call
0506 15:14:44.637742 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.638914 (+ 1172us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.639097 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.640555 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:44.641203 20878 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:44.641966 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1216) took 5468 ms. Trace:
I20240506 15:14:44.642339 20878 rpcz_store.cc:276] 0506 15:14:39.173695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.640408 (+5466713us) service_pool.cc:225] Handling call
0506 15:14:44.640831 (+ 423us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.641640 (+ 809us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.641864 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.643965 20878 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.644014 21026 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.644651 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1217) took 5470 ms. Trace:
I20240506 15:14:44.645073 20878 rpcz_store.cc:276] 0506 15:14:39.176725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.643233 (+5466508us) service_pool.cc:225] Handling call
0506 15:14:44.643589 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.644431 (+ 842us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.644600 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.646541 22232 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240506 15:14:44.646893 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 92083.7us
W20240506 15:14:44.648216 20924 leader_election.cc:334] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.823s (SENT)
I20240506 15:14:44.648926 20924 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:44.649807 22331 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:44.650321 22331 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:44.646548 20878 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.651505 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1219) took 5474 ms. Trace:
I20240506 15:14:44.651871 20878 rpcz_store.cc:276] 0506 15:14:39.177468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.645854 (+5468386us) service_pool.cc:225] Handling call
0506 15:14:44.646228 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.651194 (+ 4966us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.651417 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.653353 22418 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20240506 15:14:44.653810 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 65) took 5440 ms. Trace:
I20240506 15:14:44.632529 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1200) took 5475 ms. Trace:
I20240506 15:14:44.650406 22420 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:44.656641 20882 rpcz_store.cc:276] 0506 15:14:39.157091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.545469 (+5388378us) service_pool.cc:225] Handling call
0506 15:14:44.562093 (+ 16624us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.608619 (+ 46526us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.630317 (+ 21698us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.657145 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:44.657691 20878 rpcz_store.cc:276] 0506 15:14:39.213125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.652843 (+5439718us) service_pool.cc:225] Handling call
0506 15:14:44.653236 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.653457 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.653482 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.653518 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:44.653565 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.653615 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.653729 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.661517 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 99576.2us
I20240506 15:14:44.620541 22393 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:44.632683 22264 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:44.669891 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1212) took 5509 ms. Trace:
I20240506 15:14:44.658560 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 66) took 5321 ms. Trace:
W20240506 15:14:44.735313 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 69341.1us
I20240506 15:14:44.770644 20878 raft_consensus.cc:1261] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:44.716535 20876 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.772235 20880 rpcz_store.cc:276] 0506 15:14:39.171901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.627190 (+5455289us) service_pool.cc:225] Handling call
0506 15:14:44.627569 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.663199 (+ 35630us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.663225 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.669500 (+ 6275us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:44.669576 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.669660 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.669822 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31852,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":41206}
I20240506 15:14:44.670910 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.672068 20874 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:44.682768 20877 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.697449 20881 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.741036 20845 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:44.799814 22428 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240506 15:14:44.832110 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1208) took 5669 ms. Trace:
I20240506 15:14:44.832487 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 67) took 5489 ms. Trace:
I20240506 15:14:44.662804 20875 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.832722 20874 rpcz_store.cc:276] 0506 15:14:39.159250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.620820 (+5461570us) service_pool.cc:225] Handling call
0506 15:14:44.664009 (+ 43189us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.828546 (+164537us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.828896 (+ 350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.833514 22331 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.834955 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 94048.7us
I20240506 15:14:44.716791 22393 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:44.836575 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 71) took 3946 ms. Trace:
I20240506 15:14:44.838605 21030 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:44.741303 22338 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:44.840649 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.848762 20881 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:44.662973 22231 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:44.796217 22251 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:44.773692 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 125326us
I20240506 15:14:44.841857 20878 rpcz_store.cc:276] 0506 15:14:39.342352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.658641 (+5316289us) service_pool.cc:225] Handling call
0506 15:14:44.770227 (+111586us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.826933 (+ 56706us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.831371 (+ 4438us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.831946 22354 consensus_meta.cc:287] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.718s user 0.009s sys 0.004s
I20240506 15:14:44.798866 20879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 872) took 6281 ms. Trace:
I20240506 15:14:44.844628 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 68) took 5453 ms. Trace:
I20240506 15:14:44.836658 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.840847 20996 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:44.847883 20877 rpcz_store.cc:276] 0506 15:14:40.890224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.832617 (+3942393us) service_pool.cc:225] Handling call
0506 15:14:44.835901 (+ 3284us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.836125 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.836164 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.836218 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:44.836274 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.836332 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.836470 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.850597 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.851341 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1214) took 5678 ms. Trace:
I20240506 15:14:44.826114 20882 rpcz_store.cc:276] 0506 15:14:39.337035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.657587 (+5320552us) service_pool.cc:225] Handling call
0506 15:14:44.657941 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.658153 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.658194 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.658240 (+ 46us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:44.658305 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.658375 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.658502 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.853207 20880 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 3. Ops: []
I20240506 15:14:44.849977 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 72) took 3615 ms. Trace:
I20240506 15:14:44.762562 20780 catalog_manager.cc:5484] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 3 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:44.845278 21023 raft_consensus.cc:1226] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:44.860980 20876 rpcz_store.cc:276] 0506 15:14:39.391360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.829641 (+5438281us) service_pool.cc:225] Handling call
0506 15:14:44.844061 (+ 14420us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.844235 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.844300 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.844346 (+ 46us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:44.844380 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.844454 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.844576 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.860481 21029 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:44.863885 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 877) took 5348 ms. Trace:
I20240506 15:14:44.862630 20877 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.863783 20881 rpcz_store.cc:276] 0506 15:14:41.234976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.834444 (+3599468us) service_pool.cc:225] Handling call
0506 15:14:44.846980 (+ 12536us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.849801 (+ 2821us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.849937 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.864934 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 900) took 1537 ms. Trace:
I20240506 15:14:44.869334 20880 rpcz_store.cc:276] 0506 15:14:39.515455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.833456 (+5318001us) service_pool.cc:225] Handling call
0506 15:14:44.853018 (+ 19562us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.863616 (+ 10598us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.863833 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.869839 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1264) took 1397 ms. Trace:
I20240506 15:14:44.871060 20877 rpcz_store.cc:276] 0506 15:14:43.472229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.862031 (+1389802us) service_pool.cc:225] Handling call
0506 15:14:44.862378 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.869502 (+ 7124us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.869771 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.871519 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1209) took 5712 ms. Trace:
I20240506 15:14:44.869825 20874 rpcz_store.cc:276] 0506 15:14:43.327595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.844888 (+1517293us) service_pool.cc:225] Handling call
0506 15:14:44.850037 (+ 5149us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.864638 (+ 14601us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.864667 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.864710 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:44.864735 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.864793 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.864887 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:14:44.876212 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 276210us
I20240506 15:14:44.872961 20780 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 3 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:44.856760 22271 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:44.872315 20883 rpcz_store.cc:276] 0506 15:14:39.159504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.616034 (+5456530us) service_pool.cc:225] Handling call
0506 15:14:44.644551 (+ 28517us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.871185 (+226634us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.871219 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:44.871268 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:44.871298 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.871360 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.871464 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18554,"raft.queue_time_us":216045,"thread_start_us":197081,"threads_started":1}
W20240506 15:14:44.881016 20924 leader_election.cc:334] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.843s (SENT)
I20240506 15:14:44.881593 20924 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:44.882035 20877 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:14:44.884354 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.884887 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:44.860217 22354 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.887856 22271 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.890085 22331 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:44.890585 22271 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:44.890954 22331 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:44.891326 22331 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:44.891707 22331 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.893318 22354 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:44.894486 20845 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:44.896420 22271 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:44.896714 20924 leader_election.cc:334] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.856s (SENT)
W20240506 15:14:44.896906 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f5a9099e10624a769e45882e9b864930, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.897467 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f5a9099e10624a769e45882e9b864930, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.897774 20924 leader_election.cc:334] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.885s (SENT)
I20240506 15:14:44.898309 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:44.899720 20924 leader_election.cc:334] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.894s (SENT)
I20240506 15:14:44.900478 22271 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:44.900694 22434 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:44.901084 22434 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:44.901463 22434 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.859635 20879 rpcz_store.cc:276] 0506 15:14:38.517209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.142958 (+5625749us) service_pool.cc:225] Handling call
0506 15:14:44.172525 (+ 29567us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:44.217243 (+ 44718us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:44.217298 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:14:44.218279 (+ 981us) log.cc:842] Serialized 52 byte log entry
0506 15:14:44.366014 (+147735us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:44.542982 (+176968us) log.cc:842] Serialized 14 byte log entry
0506 15:14:44.558681 (+ 15699us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.561583 (+ 2902us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:44.626336 (+ 64753us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:44.660822 (+ 34486us) raft_consensus.cc:1671] finished
0506 15:14:44.697125 (+ 36303us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:44.798604 (+101479us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b54000bbb50. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eec59 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:44.798757 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":53365,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":25,"raft.queue_time_us":8925,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":3729,"spinlock_wait_cycles":71987456,"thread_start_us":238738,"threads_started":2,"wal-append.queue_time_us":260559}
I20240506 15:14:44.904309 20882 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.904520 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:44.904726 21027 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.905018 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 901) took 1351 ms. Trace:
I20240506 15:14:44.905438 20882 rpcz_store.cc:276] 0506 15:14:43.553501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.903499 (+1349998us) service_pool.cc:225] Handling call
0506 15:14:44.903897 (+ 398us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.904737 (+ 840us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.904960 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.908632 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:44.909601 22393 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:44.862905 20875 rpcz_store.cc:276] 0506 15:14:39.173083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.639603 (+5466520us) service_pool.cc:225] Handling call
0506 15:14:44.639971 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.842711 (+202740us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.851281 (+ 8570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.913771 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:44.914909 20952 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.874596 22403 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:44.872159 22338 raft_consensus.cc:2781] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:44.917099 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:44.917384 22338 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:44.917804 22338 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:44.917904 20953 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:44.918387 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43748.2us
I20240506 15:14:44.916694 20845 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:44.900166 20924 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:44.883252 20780 catalog_manager.cc:5484] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 2 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:44.931265 20878 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.932211 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1263) took 1473 ms. Trace:
I20240506 15:14:44.932704 20878 rpcz_store.cc:276] 0506 15:14:43.458907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.865555 (+1406648us) service_pool.cc:225] Handling call
0506 15:14:44.927964 (+ 62409us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.931871 (+ 3907us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:44.932148 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:44.938191 20780 catalog_manager.cc:5484] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 3 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:44.945948 20780 catalog_manager.cc:5484] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:44.955274 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:44.956241 20954 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:44.957676 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:44.958144 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 74536b2ff8754c03a925c7b726433266, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:44.958513 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 74536b2ff8754c03a925c7b726433266, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:44.958820 22310 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:44.960080 20846 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:44.951906 22403 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:44.974870 20876 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:44.951943 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:44.977066 20949 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:44.967185 22403 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:44.984650 20996 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:44.997025 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 97347.4us
W20240506 15:14:45.001070 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64188.3us
W20240506 15:14:45.002435 22439 log.cc:924] Time spent T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20240506 15:14:44.994031 22403 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:45.008798 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:44.965592 20780 catalog_manager.cc:5484] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:44.996804 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.011476 20780 catalog_manager.cc:5484] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:45.017227 22191 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:45.017709 22191 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.018087 22191 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:45.021378 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.029536 20952 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:44.989082 22231 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:45.034358 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:45.238039 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 225341us
I20240506 15:14:45.238734 20954 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.210702 20949 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:45.074916 20958 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
W20240506 15:14:45.079743 20998 leader_election.cc:334] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.550s (SENT)
I20240506 15:14:45.036566 20953 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.105317 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.231789 22338 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.244451 22444 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:45.244836 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.245236 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.059547 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.075770 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:45.089315 22443 log.cc:924] Time spent T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.068s user 0.002s sys 0.000s
I20240506 15:14:45.099229 22191 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.253422 20924 leader_election.cc:334] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.924s (SENT)
W20240506 15:14:45.254814 20924 leader_election.cc:334] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.925s (SENT)
I20240506 15:14:45.255311 22191 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:45.050570 22232 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:45.255332 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:45.093343 22331 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:45.256865 22393 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:45.258383 22331 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:45.258401 20924 leader_election.cc:334] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.949s (SENT)
W20240506 15:14:45.262715 20924 leader_election.cc:334] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.082s (SENT)
I20240506 15:14:45.263865 20924 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:45.266697 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 192007us
I20240506 15:14:45.263140 20954 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.256007 22232 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:45.267313 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 162129us
I20240506 15:14:45.282418 22191 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:45.283183 22191 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:45.282687 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 301517us
I20240506 15:14:45.284688 22191 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.285836 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.286760 22354 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:45.289242 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:45.290485 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:45.291538 22331 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:45.291947 22331 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.292416 22331 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:45.294138 20998 leader_election.cc:334] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.594s (SENT)
W20240506 15:14:45.295323 20998 leader_election.cc:334] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.739s (SENT)
I20240506 15:14:45.050791 22435 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:45.300701 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:45.304998 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.219740 22403 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.298975 22232 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.279577 22444 raft_consensus.cc:2741] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240506 15:14:45.095283 22434 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:45.309568 20845 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:45.310827 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b51ffbf114654032ba2038e989eec2bc, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.311293 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b51ffbf114654032ba2038e989eec2bc, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.315661 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 53109.4us
I20240506 15:14:45.316776 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:45.317305 22434 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:45.317728 20924 leader_election.cc:334] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.083s (SENT)
W20240506 15:14:45.318720 21656 leader_election.cc:334] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.810s (SENT)
I20240506 15:14:45.319363 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:45.304680 22310 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:45.306450 22435 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:45.324703 22251 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:45.320305 22444 raft_consensus.cc:2741] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240506 15:14:45.324205 22403 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:45.324909 22435 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.308698 22354 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240506 15:14:45.338585 20924 leader_election.cc:334] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.113s (SENT)
W20240506 15:14:45.344786 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49525.6us
I20240506 15:14:45.346066 22338 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:45.386682 20998 leader_election.cc:334] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.814s (SENT)
W20240506 15:14:45.391867 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47989.9us
I20240506 15:14:45.392874 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.394325 22310 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:45.394735 22310 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.394621 22420 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:45.395318 22310 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:45.401537 20845 leader_election.cc:341] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913): Illegal state: must be running to vote when last-logged opid is not known
I20240506 15:14:45.402096 20845 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:45.402957 22403 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:45.407003 22191 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.408633 22310 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.411504 22449 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20240506 15:14:45.413915 20998 leader_election.cc:334] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.829s (SENT)
I20240506 15:14:45.414439 20998 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.415262 22365 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:45.419495 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 89277.6us
I20240506 15:14:45.414155 22310 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:45.421324 20924 leader_election.cc:334] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 2.209s (SENT)
I20240506 15:14:45.410370 22448 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:45.424362 22448 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.424801 22448 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:45.339591 22435 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:45.444552 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 48450.9us
I20240506 15:14:45.446707 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.447769 22310 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:45.461386 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 931) took 1225 ms. Trace:
I20240506 15:14:45.463039 21024 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:45.465816 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.464860 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.453202 22191 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:45.462132 20874 rpcz_store.cc:276] 0506 15:14:44.235655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.872685 (+637030us) service_pool.cc:225] Handling call
0506 15:14:44.881531 (+ 8846us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:44.950121 (+ 68590us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:44.966907 (+ 16786us) log.cc:842] Serialized 14 byte log entry
0506 15:14:45.461147 (+494240us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:45.461183 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:45.461211 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:45.461258 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:45.461342 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":22553,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":2338,"raft.queue_time_us":15356,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":8245,"thread_start_us":462508,"threads_started":2,"wal-append.queue_time_us":374188}
I20240506 15:14:45.468359 21031 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:45.469543 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.461181 22310 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:45.470177 20957 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:45.473124 22393 consensus_queue.cc:1046] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:45.473975 22420 raft_consensus.cc:3032] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:45.474431 22420 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:45.475473 22420 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.460347 22403 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240506 15:14:45.476760 22448 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.477234 22420 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:45.485395 22435 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.486027 20846 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.487643 22435 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:45.398209 22331 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.479130 22448 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:45.495020 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.468118 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.489645 22310 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:45.499245 20957 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:45.498163 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.515048 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.517320 22434 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:45.517869 22434 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:45.525444 20954 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:45.520176 22435 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:45.529299 20998 leader_election.cc:334] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.870s (SENT)
I20240506 15:14:45.532223 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.532298 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.494920 22444 consensus_queue.cc:1046] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240506 15:14:45.535187 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.536063 21024 raft_consensus.cc:2379] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.536471 21023 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.537614 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.537770 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.538365 20957 raft_consensus.cc:2379] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.530005 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:45.547559 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.548898 21031 raft_consensus.cc:2379] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.499660 22393 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240506 15:14:45.554924 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.555997 20951 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:45.560387 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.562980 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.518913 22434 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:45.564497 21654 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.130:40545 timed out after 1.858s (SENT) [suppressed 86 similar messages]
W20240506 15:14:45.566882 20998 leader_election.cc:334] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.893s (SENT)
I20240506 15:14:45.567461 22331 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:45.570158 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.562091 22338 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:45.576362 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.577056 20954 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:45.566025 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.569958 22434 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:45.585574 22310 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.588481 22310 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:45.588796 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.604709 21028 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.606563 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.607867 21023 raft_consensus.cc:2379] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.608829 22435 raft_consensus.cc:3032] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:45.609230 22435 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:45.609316 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.609649 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.609443 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.610397 20951 raft_consensus.cc:2396] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20240506 15:14:45.608517 22271 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:45.610138 22435 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.610503 20949 raft_consensus.cc:2396] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:45.611622 22435 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:45.572722 22232 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.612118 22271 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:45.614713 22271 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:45.615119 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.613073 20949 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:45.617008 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.618264 20954 raft_consensus.cc:2396] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:45.613099 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.611987 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.620425 20921 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:45.620831 20921 leader_election.cc:401] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 15:14:45.621515 22420 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:45.623369 20951 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:45.624301 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.624760 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.625124 20954 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.625474 20957 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:45.626184 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.617347 22403 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:45.627039 20949 raft_consensus.cc:2396] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240506 15:14:45.632735 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.633913 20953 raft_consensus.cc:2379] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.635941 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.612561 22444 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20240506 15:14:45.661185 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.663272 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.663524 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.664166 20958 raft_consensus.cc:2396] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240506 15:14:45.654798 20951 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 2.
I20240506 15:14:45.665295 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.675388 20996 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.676864 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.662499 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.686678 21031 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:45.688817 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.689014 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.689452 21029 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:45.669732 22232 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:45.642237 22271 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:45.692106 20953 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240506 15:14:45.693292 21032 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.694468 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.694561 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.696811 21027 raft_consensus.cc:2396] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:45.700515 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.701335 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.701870 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.701236 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.702244 20952 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.702517 20951 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.703145 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.703357 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.704432 22191 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:45.700699 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.714406 20996 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.716781 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.737191 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.732628 20951 raft_consensus.cc:2396] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:45.749430 20996 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.722342 22191 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:45.754350 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.755039 20952 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.756125 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.751586 22232 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:45.762476 22191 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:45.764647 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.764776 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.765902 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.766172 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.767088 21023 raft_consensus.cc:2379] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.767026 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.768507 21027 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:45.769447 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.770546 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.771575 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:45.771133 20998 consensus_peers.cc:485] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e -> Peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Couldn't send request to peer 00fe71c1675441379f5f2c7490f9e63a. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50468 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:45.775849 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.777611 20881 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.777786 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.778394 20877 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:45.779166 21032 raft_consensus.cc:2396] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:45.767959 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.784606 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.784763 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:45.785404 20877 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.786618 20996 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.788372 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:45.782634 22466 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.137s user 0.002s sys 0.000s
I20240506 15:14:45.789976 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:45.789891 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.790630 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.792955 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.791734 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.792663 20878 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.804230 20952 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.805853 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.806645 20876 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.805866 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.807477 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:45.812168 20875 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:45.829991 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:45.829454 20924 connection.cc:232] RPC call timeout handler was delayed by -1.058260206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20240506 15:14:45.838096 20924 connection.cc:232] RPC call timeout handler was delayed by -1.0522051360001115s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.41180000000000005s for a response
I20240506 15:14:45.827618 22232 raft_consensus.cc:686] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:45.834012 20874 raft_consensus.cc:2354] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
I20240506 15:14:45.836752 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.843093 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.844329 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.852936 20950 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:45.830101 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.858805 20879 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:45.858572 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.859687 21687 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:45.874475 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.828244 22476 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:45.875371 20883 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:45.879436 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:45.861593 22338 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
W20240506 15:14:45.883713 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 97108us
W20240506 15:14:45.888947 20924 leader_election.cc:334] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.726s (SENT)
I20240506 15:14:45.888777 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.890359 20877 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.891966 21024 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:14:45.889498 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 105014us
I20240506 15:14:45.907585 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.907861 21029 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:45.908381 20883 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.910914 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:45.913712 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.914708 20956 raft_consensus.cc:3037] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.890498 22354 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:45.918565 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.920318 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:45.921089 20876 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.916679 21654 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.920867 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:45.883996 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.931912 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.933678 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.925333 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:45.934783 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.935189 20877 raft_consensus.cc:2362] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3. Re-sending same reply.
I20240506 15:14:45.943320 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:45.950619 20953 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.951783 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.952086 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.952517 20957 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:45.952767 20952 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:45.953482 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.917143 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.893617 22420 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20240506 15:14:45.964553 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.954448 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:45.970646 21031 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:45.909574 22366 consensus_queue.cc:1035] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.046s
I20240506 15:14:45.979753 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:45.949185 22444 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:45.896571 22476 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:45.991595 20876 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
W20240506 15:14:45.948335 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 86812.2us
W20240506 15:14:46.016023 22481 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20240506 15:14:46.016444 22471 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:46.016984 22484 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:46.018239 20845 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:45.982831 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:46.021070 22480 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:46.021374 22476 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:45.982579 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:46.022665 20953 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.024215 20996 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.016474 22251 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:45.984376 22489 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20240506 15:14:46.024648 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.070343 20881 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:45.979429 20879 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:45.982337 22354 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.026602 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:45.987013 20958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:45.954537 20875 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.040535 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.080322 20950 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:46.045524 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:46.044170 20956 raft_consensus.cc:2454] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.041986 20878 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:45.955250 22232 consensus_queue.cc:237] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.057873 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:46.054934 22338 raft_consensus.cc:686] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:46.068987 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 44856.8us
I20240506 15:14:46.076665 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:46.064584 22484 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.078004 20883 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.066612 22444 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:46.078552 20958 raft_consensus.cc:595] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:46.077214 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 59052us
I20240506 15:14:46.090456 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.097139 20880 raft_consensus.cc:2379] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:14:46.027529 22490 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20240506 15:14:46.100943 20958 raft_consensus.cc:603] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:14:46.100862 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:46.104671 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 59250.6us
W20240506 15:14:46.129815 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 197958us
W20240506 15:14:46.136551 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 46733us
I20240506 15:14:46.141918 20996 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.081486 22480 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.142000 20998 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:46.140908 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.095552 22476 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.146955 20881 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.143090 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:46.150431 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 49665.2us
W20240506 15:14:46.103384 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 186790us
I20240506 15:14:46.139107 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:46.151576 22444 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:46.158214 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.149279 22191 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.157842 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.163398 20845 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.164662 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.165374 20881 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:46.161052 20952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:46.166435 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.149024 22476 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:46.156280 22435 consensus_meta.cc:287] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.544s user 0.016s sys 0.000s
I20240506 15:14:46.170176 20878 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:46.144549 22480 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:46.170632 20952 raft_consensus.cc:595] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:46.162523 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.178804 20879 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 2: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:46.174157 20952 raft_consensus.cc:603] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:14:46.164064 22488 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.162633 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.164400 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.183928 22480 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:46.189775 22483 log.cc:924] Time spent T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20240506 15:14:46.167836 22191 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.198390 20882 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.173056 22365 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:46.171265 22476 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.208312 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.204756 20957 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:46.209906 20957 raft_consensus.cc:595] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:46.210394 20957 raft_consensus.cc:603] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:14:46.168093 22499 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.219833 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.220552 20882 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:46.186203 22488 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:46.204386 22191 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.203953 22435 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:46.227329 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 84292.9us
I20240506 15:14:46.231541 20998 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.234280 22489 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:46.235550 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:46.224130 22494 log.cc:924] Time spent T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240506 15:14:46.168439 22354 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.239022 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.219499 22499 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.221801 22365 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:46.226047 22488 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.239748 20882 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:46.184370 21654 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:46.240810 22501 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
W20240506 15:14:46.245676 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 73283.1us
I20240506 15:14:46.245689 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.246174 20957 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e81c09e0400045b180379648eed9b219"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:46.247963 20957 raft_consensus.cc:595] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:46.248131 20874 raft_consensus.cc:2396] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240506 15:14:46.241430 20875 raft_consensus.cc:2396] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:46.240326 22489 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.248967 22489 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.249683 22489 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.251330 22496 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:46.251812 22271 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:46.253544 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.254348 20883 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:46.254202 22465 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240506 15:14:46.255905 22503 log.cc:924] Time spent T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20240506 15:14:46.256421 22420 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:46.258327 22420 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:46.260516 22469 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:46.262471 22469 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:46.200196 22338 consensus_queue.cc:237] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.271819 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.221630 22484 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.281284 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.282691 22477 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240506 15:14:46.243070 22499 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.291285 20882 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 1.
I20240506 15:14:46.293748 22460 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240506 15:14:46.303155 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.264133 22469 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240506 15:14:46.307287 20957 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:46.307874 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.264469 22393 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240506 15:14:46.315001 20957 raft_consensus.cc:595] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:46.316717 20950 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:46.244374 22365 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:46.318773 20882 raft_consensus.cc:2396] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240506 15:14:46.316112 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.322579 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.323593 20883 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:46.324982 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.326314 20880 raft_consensus.cc:2396] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240506 15:14:46.311738 22502 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
W20240506 15:14:46.301499 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 128984us
I20240506 15:14:46.336529 22476 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.338500 22498 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:46.338739 22476 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:46.339071 22498 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:46.341037 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.341318 20949 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:46.341718 20882 raft_consensus.cc:2362] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6. Re-sending same reply.
I20240506 15:14:46.342693 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:14:46.343719 22489 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.281102 22472 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240506 15:14:46.345811 22489 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.349961 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.350356 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.353089 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.353503 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.353861 20952 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:46.354873 20957 raft_consensus.cc:2354] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
W20240506 15:14:46.354818 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 48719.1us
I20240506 15:14:46.352799 20876 raft_consensus.cc:2932] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:46.360224 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.361023 20882 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:46.361304 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.341080 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:46.362818 22495 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:46.362713 21687 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:46.363157 20953 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20240506 15:14:46.362766 22505 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:46.363737 22251 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:46.364544 20996 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.365545 22508 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:46.365751 22341 raft_consensus.cc:2781] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:46.363653 20953 pending_rounds.cc:76] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 6
I20240506 15:14:46.362751 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.366757 20953 pending_rounds.cc:97] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240506 15:14:46.371793 20953 raft_consensus.cc:2866] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:46.311986 22480 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.381753 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.356595 22489 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:46.390925 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:46.392696 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:46.348982 22477 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240506 15:14:46.375769 20779 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 3 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 6 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:46.380956 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:14:46.366451 22341 raft_consensus.cc:686] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.398675 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.398562 20882 raft_consensus.cc:2354] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:46.358961 22310 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20240506 15:14:46.413703 20880 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:46.427910 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.373823 22251 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:46.439265 20954 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:46.441681 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.451407 21031 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:46.428985 21025 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:46.426913 22456 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:46.466949 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.475497 20998 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.430563 22508 raft_consensus.cc:686] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.482021 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.482736 20882 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
W20240506 15:14:46.486079 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 223273us
I20240506 15:14:46.491873 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.492794 20883 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:46.494130 20998 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.496754 22341 consensus_queue.cc:237] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.483834 22505 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.501016 22480 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.502277 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.502628 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.503129 20875 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:46.503162 20874 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:46.503459 20879 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:46.505010 20875 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240506 15:14:46.505303 22251 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:46.516825 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 151049us
I20240506 15:14:46.499262 22488 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.481153 22338 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:46.532090 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.483314 22498 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.549842 20882 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:46.571400 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.574980 21028 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:46.485549 22456 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:46.582800 20957 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:46.540232 22480 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:46.564972 22365 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:46.531193 21656 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:46.588145 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.578290 22354 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:46.593541 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:46.594237 20949 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:46.573514 22498 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20240506 15:14:46.595135 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:46.577775 22338 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:46.595993 22354 raft_consensus.cc:686] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.584883 22480 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.596947 22480 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:46.597204 22354 consensus_queue.cc:237] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.597939 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.598555 20956 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:46.596058 22338 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.600595 22338 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:46.600770 22512 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20240506 15:14:46.603325 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:46.605402 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 90969.9us
I20240506 15:14:46.582536 22489 consensus_queue.cc:1046] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
W20240506 15:14:46.619033 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 112492us
I20240506 15:14:46.638897 20956 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:46.618618 22231 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20240506 15:14:46.582857 22505 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.641470 20996 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:46.643239 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:46.609071 22508 consensus_queue.cc:237] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.655103 22456 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:46.657567 22513 log.cc:924] Time spent T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20240506 15:14:46.658564 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:46.660737 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:46.661410 20876 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:46.662923 22516 log.cc:924] Time spent T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20240506 15:14:46.665760 22191 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.667687 22518 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:46.668200 22518 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:46.668692 22518 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:46.668736 20876 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:46.670214 20876 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:46.669876 22499 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.676110 22499 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.668049 22191 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.682790 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.616422 22341 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20240506 15:14:46.688534 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45353.1us
W20240506 15:14:46.637046 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 149669us
I20240506 15:14:46.700330 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:46.787042 22525 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:46.778297 22488 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:46.796020 22530 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:46.796540 22530 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:46.796909 22530 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:46.787772 22525 raft_consensus.cc:686] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:46.800483 22310 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:46.838611 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 62) took 1177 ms. Trace:
I20240506 15:14:46.843312 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:46.884408 22471 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:46.884756 20881 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:46.884198 20954 rpcz_store.cc:276] 0506 15:14:45.661387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:45.661518 (+ 131us) service_pool.cc:225] Handling call
0506 15:14:46.838373 (+1176855us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b5400307fd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:46.838519 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1778161536}
I20240506 15:14:46.886256 20881 raft_consensus.cc:1226] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20240506 15:14:46.886801 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 186139us
I20240506 15:14:46.885031 22471 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.887526 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.852149 20883 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:46.889552 20883 raft_consensus.cc:1226] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:46.771201 22522 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:46.853581 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:46.881588 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 181391us
I20240506 15:14:46.893257 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:46.899143 20924 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:46.914041 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:46.897109 22310 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:46.917013 20998 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.921975 20954 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:46.923027 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:46.925554 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:46.926708 21031 raft_consensus.cc:2396] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:46.870016 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:46.897982 22522 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:46.956480 22522 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:46.942682 22477 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:46.943784 22540 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20240506 15:14:46.950345 22500 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:46.951072 22521 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20240506 15:14:46.921075 22517 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.137s user 0.003s sys 0.000s
I20240506 15:14:46.916008 22496 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:46.952131 22525 consensus_queue.cc:237] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.920936 22310 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:46.954923 20956 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240506 15:14:46.955061 22480 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.898269 22488 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240506 15:14:46.973134 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 59253.7us
I20240506 15:14:46.930027 21656 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
W20240506 15:14:46.909580 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 96925.9us
I20240506 15:14:46.937626 22499 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.044590 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 194) took 2997 ms. Trace:
I20240506 15:14:47.052932 20779 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 1 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 3 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.054164 22338 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:46.412663 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:14:47.078541 22310 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:47.074012 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 185011us
I20240506 15:14:47.033779 22444 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.079574 20924 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:46.931504 22471 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:46.972580 22500 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.069542 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1024) took 1293 ms. Trace:
I20240506 15:14:47.051580 20776 catalog_manager.cc:5484] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.079123 22488 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.124574 20779 catalog_manager.cc:5484] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 5 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 6 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.150638 20776 catalog_manager.cc:5484] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.085999 22499 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.154443 20776 catalog_manager.cc:5484] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.121881 22537 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:47.161109 22499 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.076969 22496 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:47.165067 22538 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20240506 15:14:47.156090 22488 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:47.167142 22338 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.149585 20955 rpcz_store.cc:276] 0506 15:14:45.776521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:45.780751 (+ 4230us) service_pool.cc:225] Handling call
0506 15:14:45.828777 (+ 48026us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:45.861997 (+ 33220us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:45.875863 (+ 13866us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:46.177300 (+301437us) spinlock_profiling.cc:243] Waited 5.51 ms on lock 0x7b2c0005b728. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630ccd1f11 00007f630cce1542 00007f630e32afc1 00007f630e2ef7da 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782
0506 15:14:46.177700 (+ 400us) log.cc:842] Serialized 26 byte log entry
0506 15:14:46.269727 (+ 92027us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:14:46.279065 (+ 9338us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:46.291237 (+ 12172us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.068854 (+777617us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b5400041da0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e27d9b1 00007f630e288470 00007f630e288418 00007f630e288398 00007f630e288353 00007f630e2875dc 00007f63142f7fd6 00007f63142f05d8 00007f630e2c45db 00007f630e2c70d0
0506 15:14:47.069265 (+ 411us) raft_consensus.cc:1671] finished
0506 15:14:47.069289 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.069454 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60001792,"thread_start_us":62241,"threads_started":1,"wal-append.queue_time_us":71735}
I20240506 15:14:47.086095 20780 rpcz_store.cc:276] 0506 15:14:44.050805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.054198 (+ 3393us) service_pool.cc:225] Handling call
0506 15:14:44.231347 (+177149us) spinlock_profiling.cc:243] Waited 936 us on lock 0x7b54000042a8. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314686641 00007f63146816ac 00007f6314249b04 00007f6314343a8e 00007f6312657a44 00007f63126579d2 00007f6312657901 00007f631265787c 00007f6312656b32 00007f630c716bcc 00007f630c716006
0506 15:14:47.044466 (+2813119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:45.506558 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:45.511257 (+ 4699us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:45.511313 (+ 56us) write_op.cc:435] Acquired schema lock
0506 15:14:45.511325 (+ 12us) tablet.cc:660] Decoding operations
0506 15:14:45.517422 (+ 6097us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:45.517532 (+ 110us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:45.517566 (+ 34us) tablet.cc:683] Acquiring locks for 7 operations
0506 15:14:45.518526 (+ 960us) tablet.cc:699] Row locks acquired
0506 15:14:45.518549 (+ 23us) write_op.cc:260] PREPARE: finished
0506 15:14:45.518723 (+ 174us) write_op.cc:270] Start()
0506 15:14:45.518895 (+ 172us) write_op.cc:276] Timestamp: P: 1715008485518659 usec, L: 0
0506 15:14:45.518921 (+ 26us) op_driver.cc:331] REPLICATION: starting
0506 15:14:45.563590 (+ 44669us) log.cc:842] Serialized 2869 byte log entry
0506 15:14:46.037608 (+474018us) op_driver.cc:444] REPLICATION: finished
0506 15:14:46.174897 (+137289us) write_op.cc:301] APPLY: starting
0506 15:14:46.175007 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:46.175274 (+ 267us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:46.175445 (+ 171us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:46.176782 (+ 1337us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:46.177055 (+ 273us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0506 15:14:46.177119 (+ 64us) write_op.cc:312] APPLY: finished
0506 15:14:46.177442 (+ 323us) log.cc:842] Serialized 58 byte log entry
0506 15:14:46.178511 (+ 1069us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:46.178712 (+ 201us) write_op.cc:454] Released schema lock
0506 15:14:46.178756 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:46.179000 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:46.179270 (+ 270us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":57429,"reported_tablets":9,"spinlock_wait_cycles":2152704,"child_traces":[["op",{"apply.queue_time_us":136948,"apply.run_cpu_time_us":3694,"apply.run_wall_time_us":4831,"num_ops":7,"prepare.queue_time_us":256589,"prepare.run_cpu_time_us":18309,"prepare.run_wall_time_us":89960,"raft.queue_time_us":138775,"raft.run_cpu_time_us":57302,"raft.run_wall_time_us":147859,"replication_time_us":517598,"thread_start_us":370051,"threads_started":4,"wal-append.queue_time_us":57104}]]}
I20240506 15:14:47.171169 22492 raft_consensus.cc:3032] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:47.122210 22508 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20240506 15:14:47.148906 22444 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.175439 22471 consensus_queue.cc:1046] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:47.176489 22456 raft_consensus.cc:3032] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:47.176858 22456 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:47.177598 22456 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.178717 22456 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.149649 22500 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.182610 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:47.184803 22518 consensus_meta.cc:287] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.515s user 0.009s sys 0.003s
I20240506 15:14:47.185237 22518 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.171607 22492 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:47.090124 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 177902us
I20240506 15:14:47.161909 20776 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.210150 20776 catalog_manager.cc:5484] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.212474 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:47.187287 22537 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:47.216089 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:47.220708 21028 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:47.222486 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:47.213552 20956 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:47.209721 22444 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.213383 21026 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240506 15:14:47.217149 22537 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:47.258194 20954 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:47.209486 22232 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20240506 15:14:47.229562 22471 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:47.232542 22518 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:47.270718 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:47.271612 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:47.241289 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:47.280100 22530 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:47.281555 22543 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20240506 15:14:47.266968 22496 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.269212 22471 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:47.264297 22444 consensus_queue.cc:1046] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240506 15:14:47.312062 21031 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:47.306892 22518 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.317448 22518 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.319993 22518 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:47.311069 22471 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:47.309841 20956 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:14:47.297750 20776 catalog_manager.cc:5484] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.234364 22492 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.315192 22435 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:47.329196 22496 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:47.312160 22499 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.335489 22435 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:47.328096 22495 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20240506 15:14:47.339848 22492 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.330343 22231 consensus_queue.cc:1046] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240506 15:14:47.336917 20776 catalog_manager.cc:5484] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.369256 20776 catalog_manager.cc:5484] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
W20240506 15:14:47.361766 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 236885us
I20240506 15:14:47.413358 20776 catalog_manager.cc:5484] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 4, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:47.348106 22435 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:47.417744 20776 catalog_manager.cc:5484] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.433274 20776 catalog_manager.cc:5484] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.451249 20776 catalog_manager.cc:5484] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.455000 20776 catalog_manager.cc:5484] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.466738 20776 catalog_manager.cc:5484] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.503757 20776 catalog_manager.cc:5484] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.500766 20879 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
W20240506 15:14:47.481228 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 122611us
I20240506 15:14:47.523056 22480 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.522517 22310 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:47.525441 22545 log.cc:924] Time spent T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.312s user 0.002s sys 0.000s
I20240506 15:14:47.526047 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:47.527062 22310 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:47.517413 20776 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
W20240506 15:14:47.537809 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 408958us
I20240506 15:14:47.539573 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
W20240506 15:14:47.544289 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 464741us
W20240506 15:14:47.534065 22551 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20240506 15:14:47.501533 22500 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:47.549424 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 240859us
I20240506 15:14:47.551944 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:47.553056 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:47.560290 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1300) took 1056 ms. Trace:
I20240506 15:14:47.564826 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:47.545147 22530 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:47.570808 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:47.540936 22231 consensus_queue.cc:1046] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20240506 15:14:47.557153 22310 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:47.591683 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1296) took 1083 ms. Trace:
I20240506 15:14:47.593400 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1294) took 1085 ms. Trace:
I20240506 15:14:47.595072 20877 rpcz_store.cc:276] 0506 15:14:46.503175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:46.504439 (+ 1264us) service_pool.cc:225] Handling call
0506 15:14:46.504722 (+ 283us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:46.505024 (+ 302us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:46.505050 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:46.505406 (+ 356us) log.cc:842] Serialized 26 byte log entry
0506 15:14:47.078024 (+572618us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b1800268d80. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630df7e75c 00007f630b3a754c 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f630df704eb 00007f630df75f93 00007f630df76341
0506 15:14:47.078157 (+ 133us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:47.276221 (+198064us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.567708 (+291487us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.567783 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.568834 (+ 1051us) raft_consensus.cc:1671] finished
0506 15:14:47.586493 (+ 17659us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.586644 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":220077,"prepare.queue_time_us":202395,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":12396,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":31045,"spinlock_wait_cycles":3157376,"thread_start_us":327503,"threads_started":2,"wal-append.queue_time_us":115841}
I20240506 15:14:47.562429 20882 rpcz_store.cc:276] 0506 15:14:46.512606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:46.576352 (+ 63746us) service_pool.cc:225] Handling call
0506 15:14:46.585312 (+ 8960us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:46.601855 (+ 16543us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:46.601893 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:46.622978 (+ 21085us) log.cc:842] Serialized 26 byte log entry
0506 15:14:46.692119 (+ 69141us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:47.274313 (+582194us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.559902 (+285589us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.559994 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.560053 (+ 59us) raft_consensus.cc:1671] finished
0506 15:14:47.560072 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.560211 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":589677,"prepare.queue_time_us":582808,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":18267,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4198,"thread_start_us":327329,"threads_started":2,"wal-append.queue_time_us":278247}
I20240506 15:14:47.595247 20878 rpcz_store.cc:276] 0506 15:14:46.502560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:46.504369 (+ 1809us) service_pool.cc:225] Handling call
0506 15:14:46.843965 (+339596us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b58002c4998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d72e 0000000000339c39 00007f63131bc6c0 00007f63132823c4 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:46.844137 (+ 172us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:46.844605 (+ 468us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:46.844643 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:46.845170 (+ 527us) log.cc:842] Serialized 26 byte log entry
0506 15:14:47.320471 (+475301us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:47.330673 (+ 10202us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.575403 (+244730us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.575506 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.575568 (+ 62us) raft_consensus.cc:1671] finished
0506 15:14:47.587827 (+ 12259us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.588006 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":9884,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":840,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":8015744,"thread_start_us":603053,"threads_started":2,"wal-append.queue_time_us":586923}
I20240506 15:14:47.604698 20956 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:47.560994 22559 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:14:47.572935 22499 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.623523 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:47.594875 22522 consensus_meta.cc:287] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.638s user 0.005s sys 0.006s
I20240506 15:14:47.619346 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1299) took 1115 ms. Trace:
I20240506 15:14:47.564299 22534 consensus_queue.cc:1046] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20240506 15:14:47.572592 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
W20240506 15:14:47.636111 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 65409.4us
I20240506 15:14:47.644901 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:47.646456 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.563010 22488 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.606307 22574 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 15:14:47.631755 20874 rpcz_store.cc:276] 0506 15:14:46.504027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:47.066150 (+562123us) service_pool.cc:225] Handling call
0506 15:14:47.066627 (+ 477us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:47.067091 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:47.067132 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:47.072667 (+ 5535us) log.cc:842] Serialized 26 byte log entry
0506 15:14:47.230631 (+157964us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:47.553459 (+322828us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.618982 (+ 65523us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.619081 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.619125 (+ 44us) raft_consensus.cc:1671] finished
0506 15:14:47.619136 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.619281 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48378,"prepare.queue_time_us":50433,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":2465,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":486355,"threads_started":3,"wal-append.queue_time_us":276923}
I20240506 15:14:47.611393 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:47.675617 22554 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.675721 22444 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.676348 22554 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:47.676789 22554 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.676503 22444 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:47.681138 20828 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 00fe71c1675441379f5f2c7490f9e63a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:47.684746 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:47.630239 22559 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:47.649485 22553 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20240506 15:14:47.657001 22310 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.696447 22310 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.689306 22569 raft_consensus.cc:3032] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:47.622134 22365 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240506 15:14:47.690675 22500 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.657259 22499 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240506 15:14:47.693172 22559 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.657733 22556 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.657903 22574 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:47.679365 22540 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240506 15:14:47.702793 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:47.701289 22456 consensus_meta.cc:287] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.522s user 0.011s sys 0.001s
I20240506 15:14:47.679198 22482 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.703577 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1309) took 1014 ms. Trace:
I20240506 15:14:47.697552 22488 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.726377 20880 rpcz_store.cc:276] 0506 15:14:46.689433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:47.011612 (+322179us) service_pool.cc:225] Handling call
0506 15:14:47.045815 (+ 34203us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:47.047983 (+ 2168us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:47.048136 (+ 153us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:47.050566 (+ 2430us) log.cc:842] Serialized 26 byte log entry
0506 15:14:47.276366 (+225800us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:47.588426 (+312060us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.653908 (+ 65482us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.654384 (+ 476us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.702630 (+ 48246us) raft_consensus.cc:1671] finished
0506 15:14:47.702658 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.703317 (+ 659us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4346,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":103003,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":202,"thread_start_us":597788,"threads_started":3,"wal-append.queue_time_us":305987}
I20240506 15:14:47.643884 22522 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:47.748855 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8884787c4c4b474685c7e903e8789e90, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:47.749234 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8884787c4c4b474685c7e903e8789e90, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.749274 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:47.703737 22569 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:47.736017 22500 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:47.724699 22574 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:47.765053 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80379.4us
I20240506 15:14:47.784832 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:47.786386 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:47.779436 20878 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:47.779757 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:47.789633 20881 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:47.793969 20956 raft_consensus.cc:1261] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:47.731877 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:47.797183 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 150779us
I20240506 15:14:47.732540 22482 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.798519 20998 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.808202 20874 raft_consensus.cc:1226] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:47.813115 20998 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.814914 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.763777 22558 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:47.765795 22548 raft_consensus.cc:3032] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:47.765111 22527 raft_consensus.cc:3032] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:47.822399 22444 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.823323 22569 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.764375 22366 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:14:47.772054 22522 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:47.772667 22310 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.826802 20998 leader_election.cc:398] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
W20240506 15:14:47.339105 22354 consensus_queue.cc:1159] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ecededcd854842d4867fda9fa63a4b66" responder_term: 4 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:14:47.834312 22338 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
W20240506 15:14:47.848346 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9b0fa8a9db084a51a74caa1382efc1c5, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:47.848752 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9b0fa8a9db084a51a74caa1382efc1c5, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.834434 22527 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:47.849455 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:47.850244 22527 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.729738 22556 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.851915 20957 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:47.767511 22526 raft_consensus.cc:3032] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:47.852512 22526 raft_consensus.cc:729] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:47.803987 22482 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.800840 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1301) took 1258 ms. Trace:
I20240506 15:14:47.818929 22365 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:47.846087 20998 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.770262 22435 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.832365 22500 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.770709 22471 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.836410 22548 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:47.834619 22505 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:47.863818 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.868386 20881 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:14:47.841636 22366 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:47.856741 22569 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.859056 22341 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:47.859820 22310 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:47.873605 22500 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.869509 22521 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:47.893762 22477 consensus_meta.cc:287] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.859s user 0.014s sys 0.000s
I20240506 15:14:47.917274 22477 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:47.906889 22584 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.918035 22584 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.918493 22584 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.877807 22556 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.858646 20875 rpcz_store.cc:276] 0506 15:14:46.542658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:46.575697 (+ 33039us) service_pool.cc:225] Handling call
0506 15:14:46.622670 (+ 46973us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:46.634039 (+ 11369us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:46.637019 (+ 2980us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:46.641416 (+ 4397us) log.cc:842] Serialized 26 byte log entry
0506 15:14:46.883416 (+242000us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:47.607154 (+723738us) log.cc:842] Serialized 14 byte log entry
0506 15:14:47.734656 (+127502us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:47.747010 (+ 12354us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:47.755048 (+ 8038us) raft_consensus.cc:1671] finished
0506 15:14:47.755076 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:47.785037 (+ 29961us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":390893,"prepare.queue_time_us":391636,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":30,"raft.queue_time_us":86593,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":12275,"thread_start_us":552625,"threads_started":3,"wal-append.queue_time_us":246866}
W20240506 15:14:47.923594 22492 consensus_meta.cc:287] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.563s user 0.009s sys 0.000s
I20240506 15:14:47.909195 22366 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.924641 22366 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:47.877002 22590 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:47.929374 20846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.010s sys 0.019s
I20240506 15:14:47.929504 22590 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:47.929841 20846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.010s sys 0.019s
I20240506 15:14:47.930042 22590 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 2
W20240506 15:14:47.930186 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 658547us
I20240506 15:14:47.910132 22471 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.931341 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:47.910485 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 46748.5us
I20240506 15:14:47.931968 21028 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:47.933924 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.935196 20998 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.936007 22505 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:47.914714 22435 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:47.912031 22574 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.944458 22526 consensus_queue.cc:260] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.945688 22526 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:47.947055 22574 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:47.947499 21025 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:47.949040 20998 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.950244 22227 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.950369 20998 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.908231 22341 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:47.916293 21656 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:47.877442 22534 consensus_queue.cc:1046] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20240506 15:14:47.950659 22537 consensus_meta.cc:287] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.682s user 0.009s sys 0.000s
I20240506 15:14:47.915226 22500 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:47.950723 22227 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:47.972648 21031 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:47.984861 20998 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:47.930968 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:47.986443 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 55592.4us
I20240506 15:14:47.938539 22505 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:47.962013 22232 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:47.972905 22227 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:48.006618 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 220304us
W20240506 15:14:47.951206 22522 consensus_queue.cc:1159] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7610624a16b44896b6aacea1928ab20e" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20240506 15:14:48.018153 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:48.018847 22498 consensus_queue.cc:1159] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "00fe71c1675441379f5f2c7490f9e63a" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:14:47.975693 22537 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.020068 20874 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.021495 22537 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.022461 22435 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.022977 22435 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:47.962316 22495 consensus_queue.cc:1046] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20240506 15:14:48.025123 22435 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.026914 22527 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:48.027927 22521 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:47.970197 22530 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:48.045337 21032 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.045732 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:47.962306 21028 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:48.009650 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.009896 22554 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.010419 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.011533 20876 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:47.986382 22561 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:14:48.005064 22232 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.000407 22579 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:48.004276 22505 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.004456 22525 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:48.029597 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.030368 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.001616 20881 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
W20240506 15:14:47.976420 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60262.2us
I20240506 15:14:48.035840 22593 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:47.974990 22584 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.050663 22499 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.037870 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:48.041735 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:48.097605 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 59833.4us
I20240506 15:14:48.098712 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:48.055984 20882 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.063578 20880 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.099128 21031 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:48.078099 22505 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:48.101472 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.109375 22499 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:48.077595 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70356.5us
I20240506 15:14:48.167801 20924 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.169356 20924 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.080808 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.170498 20876 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:48.048595 22471 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:48.084862 20954 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.029114 22482 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.073997 22232 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.066902 20874 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240506 15:14:48.183832 20874 pending_rounds.cc:76] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 1
I20240506 15:14:48.184180 20874 pending_rounds.cc:97] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20240506 15:14:48.184520 20874 raft_consensus.cc:2866] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:48.184687 21031 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:48.104411 22435 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:48.105188 22584 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.106369 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.178908 22444 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.207072 20879 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.207460 22444 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:48.109630 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:48.210397 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.077796 22525 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.089200 22561 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:48.110100 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.149865 22574 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:48.216997 20875 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:48.217053 22574 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.219321 22574 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:48.150838 22590 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.221319 22590 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.105192 22581 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.088992 21026 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240506 15:14:48.223618 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:48.153476 22554 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.094758 22579 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.225529 22579 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.227666 22579 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.087009 22592 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:48.228291 22592 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:48.228698 22482 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.228731 22592 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.228917 20949 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
W20240506 15:14:48.232934 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43938.1us
I20240506 15:14:48.072387 21028 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.160629 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.161761 20950 raft_consensus.cc:1226] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:48.165112 20881 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.164326 22227 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.166479 22310 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:48.165030 20883 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.180202 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:48.188220 20846 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.194801 22601 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.183012 22471 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.197322 22596 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.197924 22556 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.211186 21032 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.214278 22525 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.215535 22561 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.234681 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:48.252815 21027 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.253569 20954 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:48.213606 22444 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.238572 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.255331 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.251929 20877 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.256289 20881 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:48.240757 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.241866 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.245679 20876 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:48.232934 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:48.258352 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 78255.8us
I20240506 15:14:48.209834 22500 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.259685 22227 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.259869 20924 leader_election.cc:398] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:48.252924 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.257939 20952 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
W20240506 15:14:48.258261 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 100710us
I20240506 15:14:48.261168 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.261955 22596 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:48.247581 22482 consensus_queue.cc:1046] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240506 15:14:48.241163 22310 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.241017 22601 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.259558 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.261282 22500 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.262319 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.263640 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.282171 22499 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.262383 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.259753 21023 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.283669 22596 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.284060 22310 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.284987 21030 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:48.257954 20883 raft_consensus.cc:2379] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:48.255322 22581 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:48.238788 20882 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.286068 22581 raft_consensus.cc:686] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:48.280813 22505 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.287204 21029 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:48.287053 22581 consensus_queue.cc:237] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.287915 22505 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.284862 20924 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.285437 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.289057 22435 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:48.253381 22477 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:48.257086 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.294366 20881 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:48.291059 20924 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 2. Current term is 3.
I20240506 15:14:48.263118 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.293253 20879 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.296079 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.251082 22579 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.303084 22310 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:48.306416 22548 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.306851 20949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1051) took 1353 ms. Trace:
I20240506 15:14:48.307732 22548 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:48.307461 20949 rpcz_store.cc:276] 0506 15:14:46.953821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:46.955348 (+ 1527us) service_pool.cc:225] Handling call
0506 15:14:46.955668 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:48.240235 (+1284567us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:48.306509 (+ 66274us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540038d954. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:48.306763 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2927517440}
I20240506 15:14:48.309142 22477 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.310694 22601 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.294755 22554 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.311283 21687 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:48.273384 22471 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.268321 22232 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.312402 20875 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:48.313858 22496 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:48.314095 22482 raft_consensus.cc:3032] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.299049 22569 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.315979 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.317956 22592 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.319835 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.320047 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.320330 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.370085 21024 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.370409 21028 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:48.321125 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.322413 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.323385 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.302217 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.326179 22341 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:48.326265 22227 consensus_queue.cc:1046] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:48.327884 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.311926 21031 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:48.377857 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.378106 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.378491 21031 raft_consensus.cc:2362] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4. Re-sending same reply.
W20240506 15:14:48.378751 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 55453.7us
I20240506 15:14:48.330022 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.365691 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:48.379206 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.372599 21029 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:48.363449 22365 consensus_queue.cc:1046] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:48.379259 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.379639 21027 raft_consensus.cc:2362] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5. Re-sending same reply.
I20240506 15:14:48.374012 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:48.379913 21030 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:48.387255 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.390473 20846 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.356894 22482 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.392217 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.396817 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.399827 21027 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:48.400674 21032 raft_consensus.cc:1226] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.401824 21031 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:48.403051 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.409951 21030 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.411387 21031 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.413125 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.413873 20874 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.362262 22554 consensus_queue.cc:1046] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240506 15:14:48.417542 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.420708 20880 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.422163 21029 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:48.426136 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.400833 22579 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.438138 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:48.439440 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.439956 20880 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:48.440598 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.441186 20877 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.414578 22557 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:48.447698 20877 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.447722 20880 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:48.457217 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.454175 20952 raft_consensus.cc:1203] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20240506 15:14:48.458133 20952 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:48.458184 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.460242 20880 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:48.461700 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet dae00aa7ed8440db954bef879396844e, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:48.462180 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet dae00aa7ed8440db954bef879396844e, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.466889 20846 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.410723 22590 consensus_queue.cc:1046] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240506 15:14:48.425017 22598 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:48.448503 22568 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:48.485877 20846 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:48.472244 22608 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20240506 15:14:48.398854 22477 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.419515 22596 consensus_queue.cc:1046] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
W20240506 15:14:48.512459 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 95941.9us
W20240506 15:14:48.495936 22597 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.165s user 0.001s sys 0.001s
I20240506 15:14:48.448114 22556 consensus_queue.cc:1046] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20240506 15:14:48.465690 22574 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240506 15:14:48.534265 20880 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.510087 22568 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:48.452353 22482 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.539075 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.526676 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:48.485409 22492 consensus_queue.cc:1046] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
W20240506 15:14:48.571888 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 86122.5us
I20240506 15:14:48.572821 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.535306 22232 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.538203 22579 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.539775 22592 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.539968 22568 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:48.585451 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:48.589756 20924 connection.cc:232] RPC call timeout handler was delayed by -1.2841497519999621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
I20240506 15:14:48.560493 22471 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.591081 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.601588 20998 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.565339 22488 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.607357 20953 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:48.607161 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.565724 22554 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:48.608877 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.579720 22482 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.575268 22559 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240506 15:14:48.617895 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:48.626938 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.600301 22568 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.612779 22618 raft_consensus.cc:3032] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.609457 22500 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:48.666563 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 48781.9us
W20240506 15:14:48.671442 22613 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20240506 15:14:48.671721 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.665438 20950 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:48.669620 22508 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:48.672372 22617 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.605034 22232 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240506 15:14:48.615414 22338 consensus_queue.cc:1046] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20240506 15:14:48.668732 22496 raft_consensus.cc:3032] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.628182 22624 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:48.617348 22554 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.672683 22525 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.709972 20874 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:48.664805 22354 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20240506 15:14:48.670945 22492 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:48.668644 22618 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.624413 22601 consensus_queue.cc:1046] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
W20240506 15:14:48.721053 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 112264us
I20240506 15:14:48.695732 22489 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240506 15:14:48.754653 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.705108 22620 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240506 15:14:48.728168 22625 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.730652 21686 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240506 15:14:48.732642 22570 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:48.734076 22621 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20240506 15:14:48.701048 22611 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.161s user 0.001s sys 0.001s
I20240506 15:14:48.744410 22619 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:48.748525 22554 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.748792 22471 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:48.750368 22557 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:48.750016 22624 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:48.753991 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 82344.8us
I20240506 15:14:48.694646 22617 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.759497 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.697599 22488 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
W20240506 15:14:48.763119 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 249422us
I20240506 15:14:48.766847 21025 raft_consensus.cc:1226] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.699784 22496 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.699842 22590 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:48.765481 21686 pending_rounds.cc:76] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Aborting all ops after (but not including) 6
I20240506 15:14:48.696498 22508 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.803153 22629 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:48.804641 22617 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:48.805946 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 48796.9us
I20240506 15:14:48.794473 22557 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:48.810585 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.794076 22480 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:48.819054 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.823355 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.825287 20846 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.790695 22598 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.798610 22596 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.815502 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:48.797468 22624 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:48.812754 22435 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.814711 21686 pending_rounds.cc:97] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240506 15:14:48.807554 22496 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.795044 22471 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.809505 22618 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.813227 22590 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.818149 22554 consensus_queue.cc:1046] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240506 15:14:48.819697 22530 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20240506 15:14:48.826812 20846 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.856683 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.865752 22630 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:14:48.869868 20874 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:14:48.870146 22525 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.870995 20958 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:48.866762 20846 leader_election.cc:398] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:48.868311 22616 raft_consensus.cc:3032] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:48.872571 20846 leader_election.cc:401] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240506 15:14:48.825793 20998 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.811094 22619 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.817273 22488 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.875383 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:48.876089 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 50405.3us
I20240506 15:14:48.877367 20954 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:14:48.877503 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.878067 20954 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:48.878268 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 62899.6us
I20240506 15:14:48.878757 20998 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.878964 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:48.820413 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:48.849623 22617 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.824230 22191 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:14:48.845045 22492 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:48.829512 22570 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.819240 22620 consensus_queue.cc:1046] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20240506 15:14:48.881961 22625 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.855629 21026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 110) took 1156 ms. Trace:
I20240506 15:14:48.831442 22629 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.874114 22596 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.890640 20952 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.891009 20957 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:48.893456 22618 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.910045 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.910494 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.883268 20954 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.873769 21686 raft_consensus.cc:2866] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:48.873970 22616 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:48.877373 22630 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:48.856146 22508 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.901655 22632 consensus_queue.cc:1046] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:48.902837 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.885205 22525 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:48.882961 22559 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.903226 22499 raft_consensus.cc:3032] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:48.880496 22484 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:48.880066 22558 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:48.930645 22568 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.912058 20954 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:48.937369 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.823213 22626 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.085s
I20240506 15:14:48.893962 22624 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:48.916692 20958 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:48.904686 22496 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.920852 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.905642 22492 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.924443 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.926096 22484 raft_consensus.cc:686] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.957906 22471 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.924575 22499 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.960604 22616 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.926838 22488 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.912348 22635 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.961690 22499 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.964129 22635 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:48.964913 22614 log.cc:924] Time spent T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.453s user 0.003s sys 0.000s
I20240506 15:14:48.965173 22635 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.966585 22635 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:48.966851 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.968035 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:48.969059 21686 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
W20240506 15:14:48.969615 22639 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240506 15:14:48.830346 22480 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.971937 22480 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:48.942191 22622 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:48.941718 20957 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:48.974094 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.974586 22559 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:48.975116 22559 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.975205 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.944380 20956 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:14:48.944383 20952 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:48.976059 22559 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.964922 22499 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:48.979096 20952 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.979585 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:48.962038 22616 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:48.982744 22557 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:48.969115 22341 raft_consensus.cc:3032] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:48.985474 22341 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.986270 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.944922 22641 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:48.945673 20998 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.945793 22505 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:48.999037 22620 raft_consensus.cc:3032] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:48.946493 22568 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.947146 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:48.947376 22640 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:48.947743 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:48.948083 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.948429 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.002929 20878 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:48.949558 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:48.949801 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.950129 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.951169 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:48.951511 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:48.952220 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.904834 21026 rpcz_store.cc:276] 0506 15:14:47.699108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:47.707022 (+ 7914us) service_pool.cc:225] Handling call
0506 15:14:47.766244 (+ 59222us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:48.223118 (+456874us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:48.855156 (+632038us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b540015f4d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:48.855512 (+ 356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":694069504}
I20240506 15:14:48.957441 22484 consensus_queue.cc:237] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.931872 22598 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:48.968183 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:48.969009 22590 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:48.935858 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.938580 22525 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:48.895787 22561 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20240506 15:14:48.986707 20952 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:48.988631 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.991567 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:48.994290 22489 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.030424 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:48.994421 22341 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.035689 22341 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:48.996366 22638 raft_consensus.cc:3032] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.036412 22638 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.036674 22568 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:48.996876 22521 raft_consensus.cc:3032] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:48.997385 20950 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:49.004642 20874 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:49.014230 20880 raft_consensus.cc:1226] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:49.016762 20958 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:49.016794 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 78698.8us
I20240506 15:14:49.018987 20877 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240506 15:14:49.019587 20957 raft_consensus.cc:1226] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:49.001190 22620 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.027334 20954 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
W20240506 15:14:49.028995 22633 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.179s user 0.002s sys 0.000s
I20240506 15:14:49.030740 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.030702 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:49.016371 22640 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.022611 22601 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.045584 20954 pending_rounds.cc:76] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 6
I20240506 15:14:49.045914 20952 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:49.048434 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.044266 20922 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:49.050143 20878 raft_consensus.cc:2362] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5. Re-sending same reply.
W20240506 15:14:49.052374 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 88365.3us
I20240506 15:14:49.054057 22636 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.054121 22548 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.051520 22505 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240506 15:14:49.059978 20998 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.054612 22489 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:49.055348 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:49.057510 22609 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.547s user 0.002s sys 0.000s
I20240506 15:14:49.061213 20949 raft_consensus.cc:2396] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240506 15:14:49.046613 22488 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:49.108748 22616 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:49.061419 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.060019 20954 pending_rounds.cc:97] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240506 15:14:49.037643 22521 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.109491 20954 raft_consensus.cc:2866] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:49.110787 20953 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:49.106943 20958 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:49.037191 22638 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.109745 20950 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:49.065999 21654 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:14:49.115551 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55650.1us
I20240506 15:14:49.116477 20998 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:49.116544 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 50663.8us
I20240506 15:14:49.117440 22642 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.117919 22642 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.119575 22642 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.120934 22569 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240506 15:14:49.121104 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 59796.2us
I20240506 15:14:49.121688 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.121830 20846 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.122660 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.122948 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.123106 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.123703 20958 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 2: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:49.123737 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.121893 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:48.944268 22596 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.170181 20950 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:49.131359 20879 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:49.162056 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1335) took 1075 ms. Trace:
I20240506 15:14:49.124619 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:49.169799 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 53417.5us
I20240506 15:14:49.127130 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:49.227021 21025 rpcz_store.cc:276] 0506 15:14:48.086389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:48.086574 (+ 185us) service_pool.cc:225] Handling call
0506 15:14:48.099839 (+ 13265us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:48.796972 (+697133us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:49.161819 (+364847us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b540021b6d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:49.162001 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1533421440}
I20240506 15:14:49.227810 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.167174 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.124819 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.124186 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:49.230358 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 106218us
I20240506 15:14:49.231904 20846 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:49.158535 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 99735.8us
I20240506 15:14:49.232967 20956 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:49.235229 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.108958 22488 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.236325 20949 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:49.237742 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:49.137148 22610 log.cc:924] Time spent T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.636s user 0.002s sys 0.000s
I20240506 15:14:49.247737 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:49.247882 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.240933 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:49.166211 22647 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240506 15:14:49.124852 22568 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.169358 20955 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240506 15:14:49.113128 22616 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:49.174028 22338 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.248729 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:49.150934 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:49.252710 20957 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:49.165944 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.136011 22574 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:49.163601 22640 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.140965 22522 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:49.158263 22557 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:49.124466 22569 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.255918 20949 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:49.158493 22508 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.255858 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.170117 22499 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:49.058631 22584 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:49.257958 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.258476 20878 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:49.254850 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:49.251027 22505 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:49.277051 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.107980 22601 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.233204 22622 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.168231 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.289528 22634 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240506 15:14:49.261065 20952 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:49.262046 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.259519 22574 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.107064 22548 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.070431 22492 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.269732 22556 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
W20240506 15:14:49.330266 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 212750us
W20240506 15:14:49.330876 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 73017.3us
I20240506 15:14:49.255373 22338 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.255300 22616 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:49.281136 20876 raft_consensus.cc:2396] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240506 15:14:49.283110 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:49.334225 20883 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.283377 20996 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:49.284140 22505 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.289160 20879 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.263362 22581 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.262863 22579 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:49.292371 22601 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.240773 22530 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240506 15:14:49.321472 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.322644 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.375105 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.299116 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:49.320014 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44369.9us
I20240506 15:14:49.326692 20957 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:49.292642 22527 raft_consensus.cc:3032] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Stepping down as leader of term 2
I20240506 15:14:49.384472 22527 raft_consensus.cc:729] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:49.366602 22556 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.157630 22620 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.327685 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.293865 20956 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:49.330173 22499 raft_consensus.cc:686] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.297565 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:49.335952 21030 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:49.274339 22522 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:49.372709 22482 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.136981 22521 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.343773 22584 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.148703 22525 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.294772 22496 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20240506 15:14:49.340034 22488 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:49.344228 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 60910.7us
I20240506 15:14:49.410169 20998 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.345775 22232 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:49.333348 22569 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:49.371977 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.383332 22548 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.279569 22489 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20240506 15:14:49.383994 21026 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:49.393303 20846 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.400019 21025 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:49.369891 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.402679 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.433884 21028 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:49.412096 20883 raft_consensus.cc:2454] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:49.434989 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.393492 22622 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.113699 22636 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.405164 22496 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.411864 20879 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:49.279870 22617 consensus_queue.cc:1046] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
I20240506 15:14:49.414938 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:49.443214 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 49984.9us
I20240506 15:14:49.444283 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.445811 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.388203 22559 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:49.403841 22482 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:14:49.441098 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 109633us
I20240506 15:14:49.429351 22521 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:49.398643 22593 consensus_queue.cc:1046] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240506 15:14:49.471269 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.475543 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:49.475462 21653 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:49.428598 22592 consensus_queue.cc:1046] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20240506 15:14:49.438740 22619 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:49.490531 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.427765 22558 consensus_queue.cc:1046] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20240506 15:14:49.426342 22530 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240506 15:14:49.453375 22482 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.446641 22496 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.455015 22581 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:49.403981 22522 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:49.413112 22480 consensus_queue.cc:1159] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ecededcd854842d4867fda9fa63a4b66" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:14:49.470829 22310 raft_consensus.cc:3032] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:49.526422 22310 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.494557 22444 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.475809 22598 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.501324 22232 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:49.510836 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:49.403344 22505 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.402086 22499 consensus_queue.cc:237] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.512130 22648 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.539387 22648 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:49.539777 22648 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.500655 22435 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.514657 22548 consensus_queue.cc:1046] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:49.515611 22649 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20240506 15:14:49.554458 22527 consensus_queue.cc:260] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.522296 22622 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:49.401049 22574 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.395206 22556 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.404700 22584 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.453202 22559 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.478225 22586 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.538116 22617 raft_consensus.cc:3032] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.538497 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.538789 22636 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.529678 22645 raft_consensus.cc:3032] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.555433 22548 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.519306 22496 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:49.560133 22652 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20240506 15:14:49.441915 22338 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.548458 21685 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 6
I20240506 15:14:49.548089 22435 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:49.568140 20877 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:49.527446 22310 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.499609 20846 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.608671 21656 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:49.610044 22586 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 15:14:49.611431 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 111886us
I20240506 15:14:49.574345 21023 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:49.567703 22617 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.572744 22574 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:49.573823 22645 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.619478 20880 raft_consensus.cc:1226] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
W20240506 15:14:49.600513 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 190392us
I20240506 15:14:49.618268 20846 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.634951 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:49.586918 22527 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:49.637660 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:49.585980 22548 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.638638 22642 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:49.639230 22586 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.641229 20875 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:49.648562 22624 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:49.615690 22435 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.654260 20998 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.566049 22622 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.661687 21028 raft_consensus.cc:1226] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:49.665632 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.667388 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.684582 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.655012 22642 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:49.655238 22624 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.662431 22496 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.689198 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.696916 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.669085 22489 raft_consensus.cc:2781] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:49.693984 20874 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:49.660892 22658 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.678522 22586 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.700105 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.684859 22617 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.713689 22617 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.707532 22658 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:49.716837 22658 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.717324 22559 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:49.718852 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:49.723464 22310 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:49.696367 22598 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:49.692760 22579 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.691218 22642 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:49.698733 22628 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 15:14:49.700438 22620 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.734685 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:49.738458 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:49.739300 20952 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:49.740108 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.701526 22554 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.743491 20952 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:49.757858 22628 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.758332 20924 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.760226 22590 consensus_queue.cc:1046] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:49.760874 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.760658 22586 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:49.715605 22624 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.730033 22626 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.733050 22579 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:49.731909 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.762117 22489 raft_consensus.cc:686] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:49.787914 20874 raft_consensus.cc:1226] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:49.783249 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.796713 20846 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:49.798338 22489 consensus_queue.cc:237] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.789160 22641 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:49.801159 22435 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.801432 22641 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.803452 22641 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:49.804301 22548 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.807227 20878 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:49.791503 22635 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:49.808537 20878 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:49.809441 20875 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:49.811540 20875 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:49.812199 20954 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:49.775516 22590 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20240506 15:14:49.816452 22660 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20240506 15:14:49.817004 22435 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.818341 22338 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.694749 22645 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.815615 22482 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.821574 22645 raft_consensus.cc:3037] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.822889 21029 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:49.830752 20874 raft_consensus.cc:1226] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:49.831429 22648 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.770362 22558 raft_consensus.cc:3032] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:49.846267 22558 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:49.847121 22558 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:49.847481 22658 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.833180 20949 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:49.833137 20875 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:49.784422 22579 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.836156 22586 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.830376 22482 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:49.840221 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 65479.9us
I20240506 15:14:49.842773 22619 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:49.833768 22596 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.877839 21024 raft_consensus.cc:1226] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
W20240506 15:14:49.851392 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 143832us
W20240506 15:14:49.875479 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 78871.4us
I20240506 15:14:49.842890 21685 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:49.886623 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:49.851409 22558 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.860206 22636 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.907390 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.859798 22586 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:49.840728 22616 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.917510 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:49.885447 22581 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:49.946652 22581 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:49.947333 22581 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.902088 22568 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:49.949242 22568 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:49.911711 22622 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.907790 22619 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20240506 15:14:49.946022 21686 raft_consensus.cc:2354] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
W20240506 15:14:49.973907 22569 consensus_meta.cc:287] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.544s user 0.010s sys 0.000s
I20240506 15:14:49.974395 22569 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:49.975697 20880 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 2. Current term is 3. Ops: []
I20240506 15:14:49.130014 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:14:49.946583 22638 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.951074 22482 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240506 15:14:49.951983 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49371.8us
I20240506 15:14:49.953706 22616 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:49.913450 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:49.941840 21689 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:49.909511 22648 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:49.892475 22574 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:49.959753 22559 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:49.977141 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:49.942770 22586 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.999871 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.012813 20879 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:50.000136 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:49.965729 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 86397.7us
I20240506 15:14:50.009600 21030 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:50.013644 20876 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 4
I20240506 15:14:49.941834 22508 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20240506 15:14:50.007982 20951 raft_consensus.cc:2354] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6.
I20240506 15:14:50.027343 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:50.042351 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:50.039451 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:50.043501 20956 raft_consensus.cc:2379] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:50.025434 22673 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.084s user 0.000s sys 0.004s
I20240506 15:14:50.053934 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:50.062075 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 85061us
I20240506 15:14:50.065550 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.068835 22556 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:50.070025 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.070148 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.070729 21023 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:50.038765 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:50.071494 22624 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:50.072057 21684 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:50.072124 21026 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:14:50.075764 22683 log.cc:924] Time spent T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240506 15:14:50.076650 22636 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:50.077167 22677 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20240506 15:14:50.062964 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.077843 21028 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.078786 21025 raft_consensus.cc:1226] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:50.079735 22626 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:50.080353 20996 leader_election.cc:398] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:50.080665 20996 leader_election.cc:401] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:50.035387 22477 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.071535 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.083683 21024 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 2.
I20240506 15:14:50.028685 22435 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.099165 22227 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:50.087960 20879 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:50.093680 20876 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:50.095572 21028 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:50.092305 20845 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:50.013938 22574 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.018600 22559 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.039041 22618 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:50.130079 22581 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.132370 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 69337.9us
I20240506 15:14:50.147837 20996 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
W20240506 15:14:50.147976 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 55817.4us
W20240506 15:14:50.149310 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 54518.4us
I20240506 15:14:50.148766 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:50.099740 22227 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:50.077188 22636 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.055073 22617 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20240506 15:14:50.212313 22619 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:50.221527 22505 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:50.154716 22482 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.144533 22658 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.213689 20957 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240506 15:14:50.082823 22341 consensus_queue.cc:1046] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
W20240506 15:14:50.216931 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 69170.2us
W20240506 15:14:50.217592 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 65251.5us
I20240506 15:14:50.219693 22227 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.149953 22444 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.085638 21656 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
W20240506 15:14:50.211460 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57392.9us
I20240506 15:14:50.106743 22435 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.095953 22477 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.242633 22690 log.cc:924] Time spent T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20240506 15:14:50.231601 22232 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:50.132200 22310 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.135s
I20240506 15:14:50.290570 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:50.229099 22622 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
I20240506 15:14:50.276078 22617 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:50.308614 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 62630.4us
W20240506 15:14:50.299582 22496 consensus_meta.cc:287] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.596s user 0.007s sys 0.003s
I20240506 15:14:50.320971 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.322072 21028 raft_consensus.cc:2354] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4.
I20240506 15:14:50.317648 20950 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.276594 22648 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20240506 15:14:50.278126 22619 consensus_queue.cc:1046] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240506 15:14:50.297266 22232 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:50.306246 22527 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:50.333185 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.334031 20845 leader_election.cc:398] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4.
I20240506 15:14:50.358176 21028 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:14:50.302516 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 247486us
I20240506 15:14:50.310830 22557 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.360929 20956 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:14:50.365078 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 54543.9us
I20240506 15:14:50.319088 22505 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.366175 20998 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:50.365175 20846 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:50.333606 22642 consensus_meta.cc:287] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.585s user 0.004s sys 0.007s
I20240506 15:14:50.337477 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.372634 20881 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:50.363073 20845 leader_election.cc:401] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:50.370647 20950 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.320017 22581 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.340756 22232 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:50.378172 22642 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.382649 22686 log.cc:924] Time spent T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
W20240506 15:14:50.383529 22693 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240506 15:14:50.386292 20956 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:50.395809 20956 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:50.331892 22568 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:50.360527 22527 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:50.334527 22530 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.362758 22691 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.316829 22617 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.391310 22581 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4.
I20240506 15:14:50.396781 20956 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.400053 22586 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.403911 22681 log.cc:924] Time spent T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.194s user 0.001s sys 0.001s
I20240506 15:14:50.405390 22642 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.380712 22191 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240506 15:14:50.396282 22619 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:50.411648 22568 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:50.413491 22338 consensus_meta.cc:287] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.595s user 0.007s sys 0.006s
I20240506 15:14:50.473958 22568 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.413836 22444 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:50.411021 22645 consensus_meta.cc:287] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.560s user 0.007s sys 0.000s
I20240506 15:14:50.391249 22684 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:50.477888 22684 raft_consensus.cc:686] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:50.479074 22684 consensus_queue.cc:237] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.473529 22619 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:50.399113 22574 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.417338 22527 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.374058 22557 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.420511 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:50.376843 22505 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.466926 22622 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.465653 22620 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.471611 22559 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:50.505708 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 122737us
I20240506 15:14:50.519448 21032 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.422940 22530 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.457120 21656 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:50.551045 21686 raft_consensus.cc:2354] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4.
I20240506 15:14:50.507262 22574 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:50.523392 22558 consensus_meta.cc:287] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.615s user 0.004s sys 0.008s
W20240506 15:14:50.583734 22579 consensus_meta.cc:287] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.706s user 0.008s sys 0.002s
I20240506 15:14:50.584458 22581 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:50.585511 22694 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240506 15:14:50.512605 22568 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.513075 22477 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.423801 22691 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.614336 22691 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.590040 22624 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.615222 22624 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.617931 22624 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.522754 22620 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.516453 21687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59110 (request call id 176) took 1023 ms. Trace:
I20240506 15:14:50.526654 22435 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:50.603541 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 238414us
W20240506 15:14:50.597864 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 199369us
I20240506 15:14:50.636515 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:50.522998 22622 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.606264 22581 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.607815 22617 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.625222 22505 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:50.591236 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 212131us
I20240506 15:14:50.603233 22648 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election won for term 4
W20240506 15:14:50.634771 20846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.130:40545 timed out after 1.607s (SENT) [suppressed 27 similar messages]
I20240506 15:14:50.504345 22619 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.637780 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.637805 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:50.638640 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 304728us
I20240506 15:14:50.598462 22574 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.648595 21028 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:50.655267 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.656168 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.657119 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.650148 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.645416 20955 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:50.657835 21027 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:50.659526 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:50.628075 22586 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.660640 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:50.634055 22482 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:50.653446 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.662909 22568 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.663340 21024 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.664207 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.698024 22568 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.698683 21031 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 2.
I20240506 15:14:50.664996 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.665604 22477 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.700328 22477 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.701973 22700 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
W20240506 15:14:50.702373 22692 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.483s user 0.004s sys 0.001s
I20240506 15:14:50.667367 22191 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:50.703373 20951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 78) took 2108 ms (client timeout 2384 ms). Trace:
I20240506 15:14:50.703526 22557 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.703991 22706 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.703873 22191 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.704605 22706 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.706674 22191 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.707406 21030 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:50.636049 22435 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.708714 22435 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.709779 22569 raft_consensus.cc:3032] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:50.710197 22569 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:50.710692 22435 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.665262 22705 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.710999 22569 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.711498 22705 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.679880 22232 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.713665 22705 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.714848 22232 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:50.718866 20956 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:50.719444 22638 consensus_meta.cc:287] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.722s user 0.008s sys 0.000s
I20240506 15:14:50.720005 20958 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:14:50.703794 20951 rpcz_store.cc:269] 0506 15:14:48.594982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:48.600964 (+ 5982us) service_pool.cc:225] Handling call
0506 15:14:49.165755 (+564791us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b58000a2998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313231f50 00007f6313231834 00007f630c915c14 00007f630c915ba2
0506 15:14:50.703148 (+1537393us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b5400307fd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:50.703296 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616583680}
I20240506 15:14:50.720717 22496 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.721235 22496 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.659212 22617 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:50.722709 22617 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:50.723058 22617 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:50.673858 22708 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:50.686219 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 601275us
I20240506 15:14:50.724516 22708 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.688910 22622 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.627789 21687 rpcz_store.cc:276] 0506 15:14:49.481942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:49.488318 (+ 6376us) service_pool.cc:225] Handling call
0506 15:14:49.520741 (+ 32423us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:49.748958 (+228217us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:49.753501 (+ 4543us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:49.798091 (+ 44590us) log.cc:842] Serialized 26 byte log entry
0506 15:14:50.034811 (+236720us) raft_consensus.cc:1631] Marking committed up to 6
0506 15:14:50.038705 (+ 3894us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:50.038792 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:50.490494 (+451702us) raft_consensus.cc:1671] finished
0506 15:14:50.494385 (+ 3891us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:50.505433 (+ 11048us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":34035,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":433002,"threads_started":2,"wal-append.queue_time_us":433408}
I20240506 15:14:50.726781 22708 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.688752 21027 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:14:50.729120 22620 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.673089 20882 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:50.695420 22641 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.730880 22641 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:50.733745 20958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35816 (request call id 886) took 1609 ms (client timeout 1607 ms). Trace:
I20240506 15:14:50.734015 20953 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240506 15:14:50.734166 20958 rpcz_store.cc:269] 0506 15:14:49.123818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:49.124376 (+ 558us) service_pool.cc:225] Handling call
0506 15:14:50.733537 (+1609161us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540019e954. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:50.733666 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3427976960}
I20240506 15:14:50.737304 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.696782 22530 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:50.696890 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 53439.2us
I20240506 15:14:50.661762 22648 raft_consensus.cc:686] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:50.698405 22354 raft_consensus.cc:3032] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
W20240506 15:14:50.726565 21032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1361) took 1429 ms (client timeout 1548 ms). Trace:
W20240506 15:14:50.754105 21032 rpcz_store.cc:269] 0506 15:14:49.297162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:49.297303 (+ 141us) service_pool.cc:225] Handling call
0506 15:14:50.726363 (+1429060us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001acf54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:50.726483 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2468124032}
I20240506 15:14:50.694536 22636 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.603665 22579 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.702471 22477 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.747359 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 68) took 5187 ms. Trace:
W20240506 15:14:50.752166 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 301846us
I20240506 15:14:50.751545 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.762724 20881 raft_consensus.cc:2379] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:50.759027 21030 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:50.762938 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.764094 21653 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:50.764967 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.767274 21030 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.768149 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.768505 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.768867 21030 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.770363 21030 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:50.770524 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.771517 21027 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
W20240506 15:14:50.753154 22596 consensus_meta.cc:287] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.859s user 0.008s sys 0.004s
I20240506 15:14:50.773921 20881 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.774317 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.741122 22569 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.779364 20956 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.776021 20921 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:50.780635 20882 raft_consensus.cc:2379] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:50.775450 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:50.781565 22618 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240506 15:14:50.774313 20950 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:50.756747 22354 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:50.793298 20881 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:50.781795 20778 catalog_manager.cc:5484] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:50.797365 22574 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.773636 22568 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.781963 21026 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:14:50.762040 20779 rpcz_store.cc:276] 0506 15:14:45.560010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:45.560185 (+ 175us) service_pool.cc:225] Handling call
0506 15:14:45.597132 (+ 36947us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54000037d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e274d30 00007f630e2f817a 00007f6314249b1c 00007f6314343a8e 00007f6312657a44 00007f63126579d2 00007f6312657901 00007f631265787c 00007f6312656b32 00007f630c716bcc 00007f630c716006
0506 15:14:50.747274 (+5150142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:47.636965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:47.691704 (+ 54739us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:47.691742 (+ 38us) write_op.cc:435] Acquired schema lock
0506 15:14:47.691761 (+ 19us) tablet.cc:660] Decoding operations
0506 15:14:47.692325 (+ 564us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:47.692393 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:47.692420 (+ 27us) tablet.cc:683] Acquiring locks for 3 operations
0506 15:14:47.692715 (+ 295us) tablet.cc:699] Row locks acquired
0506 15:14:47.692736 (+ 21us) write_op.cc:260] PREPARE: finished
0506 15:14:47.919631 (+226895us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b2c00001368. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630ccd1f11 00007f630cce0bda 00007f630cce0863 00007f630cce0801 00007f630e32aec2 00007f630e32ac9d 00007f6311d00160 00007f6311cffc78 00007f6311d04561 00007f6311d04519 00007f6311d044a9
0506 15:14:47.919702 (+ 71us) write_op.cc:270] Start()
0506 15:14:47.919919 (+ 217us) write_op.cc:276] Timestamp: P: 1715008487704007 usec, L: 0
0506 15:14:47.919945 (+ 26us) op_driver.cc:331] REPLICATION: starting
0506 15:14:47.920636 (+ 691us) log.cc:842] Serialized 1297 byte log entry
0506 15:14:48.425571 (+504935us) op_driver.cc:444] REPLICATION: finished
0506 15:14:48.998687 (+573116us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b1800284740. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3c51b8 00007f630b3c8691 00007f630b3c8649 00007f630b3c85d9 00007f630b3c85a1 00007f630b3c789d 00007f6314639da4 00007f6314639bd9 00007f630b3a7936
0506 15:14:48.998828 (+ 141us) write_op.cc:301] APPLY: starting
0506 15:14:48.998910 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:48.999044 (+ 134us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:48.999063 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:49.029092 (+ 30029us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:49.029292 (+ 200us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0506 15:14:49.029347 (+ 55us) write_op.cc:312] APPLY: finished
0506 15:14:49.031957 (+ 2610us) log.cc:842] Serialized 34 byte log entry
0506 15:14:49.032145 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:49.032240 (+ 95us) write_op.cc:454] Released schema lock
0506 15:14:49.032301 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:49.032563 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:49.032590 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1734099,"reported_tablets":5,"spinlock_wait_cycles":80979840,"child_traces":[["op",{"apply.queue_time_us":111810,"apply.run_cpu_time_us":2786,"apply.run_wall_time_us":34010,"num_ops":3,"prepare.queue_time_us":433659,"prepare.run_cpu_time_us":12021,"prepare.run_wall_time_us":440715,"raft.queue_time_us":238009,"raft.run_cpu_time_us":29041,"raft.run_wall_time_us":334810,"replication_time_us":495652,"spinlock_wait_cycles":41247360,"thread_start_us":576718,"threads_started":4,"wal-append.queue_time_us":159805,"wal-append.run_cpu_time_us":47135,"wal-append.run_wall_time_us":1953319}]]}
I20240506 15:14:50.779348 22586 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.775462 22675 raft_consensus.cc:2781] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:50.814059 20882 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:50.815529 20881 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.782299 22556 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.799614 22596 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:50.819703 20880 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240506 15:14:50.823768 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 123494us
W20240506 15:14:50.827452 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 192690us
I20240506 15:14:50.831111 20880 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:50.790752 22634 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240506 15:14:50.835116 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:50.778623 22527 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.838356 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.835863 20846 leader_election.cc:398] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 3. Current term is 4.
I20240506 15:14:50.838784 20882 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.840382 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.840544 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.840966 20877 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:50.841082 20883 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:50.819571 22581 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.843063 20921 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.845644 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.846937 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.855729 20921 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.856645 20880 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:14:50.857556 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.858165 20880 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.859258 20880 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.853336 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.861267 20883 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.847774 20846 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.862722 20877 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:14:50.862864 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:50.828163 22675 raft_consensus.cc:686] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:50.860082 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:50.877344 21684 raft_consensus.cc:2354] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:50.836768 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:50.868877 22354 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.877620 22675 consensus_queue.cc:237] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.878727 22354 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.863408 22310 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:50.880443 22310 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:50.880832 22310 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:50.880918 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.881637 20880 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:50.874112 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.882331 20875 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.884181 22579 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.813271 22648 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:50.885373 20877 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:50.893689 20996 leader_election.cc:398] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:50.894237 20996 leader_election.cc:401] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240506 15:14:50.849607 22620 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:14:50.903795 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:50.904752 20950 raft_consensus.cc:2354] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6.
I20240506 15:14:50.827999 22556 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.910360 22556 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.822057 22586 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.854655 22646 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:50.900327 20874 raft_consensus.cc:1226] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:50.901309 22636 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.902019 22641 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.850337 22568 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.849488 22703 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.899401 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.913180 21654 leader_election.cc:398] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6.
I20240506 15:14:50.926273 21654 leader_election.cc:401] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:50.933295 22496 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.918846 22708 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.912856 20845 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:50.941922 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:50.917016 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:50.917676 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 54867.8us
I20240506 15:14:50.842092 22598 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20240506 15:14:50.923619 22622 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:50.943109 20874 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:50.947679 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.868340 22581 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:50.951825 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.953289 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.948457 20874 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:50.955345 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.956076 21032 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:50.937697 20875 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:50.956764 22527 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.956851 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.958447 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.936496 22435 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.936697 22658 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6.
I20240506 15:14:50.953912 21027 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:50.961987 22435 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.962613 22617 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.963966 22435 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.965756 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:50.947005 22579 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:50.970296 22579 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:50.970765 22579 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.971438 22717 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:50.971510 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.973141 22579 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.972400 20845 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:50.975661 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.976980 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.977699 20877 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.978134 22521 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:50.982789 22586 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.961036 22714 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:50.979463 22617 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.983695 22714 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.985793 22484 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:50.986476 22714 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.986415 22484 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.956851 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.989296 22718 consensus_queue.cc:1046] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:50.936832 22703 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.990218 22521 raft_consensus.cc:3032] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:50.990585 22521 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:50.991595 22703 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:50.991494 22718 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:50.988204 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.992987 20875 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:50.994009 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.994640 20875 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:50.995517 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.988557 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:50.996119 20875 raft_consensus.cc:3037] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:50.997818 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:50.983875 22706 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:50.998662 20874 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:50.999639 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.001775 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54786 (request call id 191) took 6758 ms. Trace:
I20240506 15:14:51.003010 20778 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:50.926401 22568 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.005321 20882 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:51.012565 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.013072 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.991396 22521 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:50.973804 22579 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.021149 20778 catalog_manager.cc:5484] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:51.022876 20875 raft_consensus.cc:2454] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:51.027946 20778 catalog_manager.cc:5484] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:51.012182 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:50.996665 22435 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:51.002213 20776 rpcz_store.cc:276] 0506 15:14:44.243138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.246559 (+ 3421us) service_pool.cc:225] Handling call
0506 15:14:51.001691 (+6755132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:48.093886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:48.123280 (+ 29394us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:48.123332 (+ 52us) write_op.cc:435] Acquired schema lock
0506 15:14:48.123358 (+ 26us) tablet.cc:660] Decoding operations
0506 15:14:48.189905 (+ 66547us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:48.189978 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:48.190022 (+ 44us) tablet.cc:683] Acquiring locks for 16 operations
0506 15:14:48.191802 (+ 1780us) tablet.cc:699] Row locks acquired
0506 15:14:48.191829 (+ 27us) write_op.cc:260] PREPARE: finished
0506 15:14:48.192027 (+ 198us) write_op.cc:270] Start()
0506 15:14:48.192167 (+ 140us) write_op.cc:276] Timestamp: P: 1715008488191963 usec, L: 0
0506 15:14:48.192200 (+ 33us) op_driver.cc:331] REPLICATION: starting
0506 15:14:48.258937 (+ 66737us) log.cc:842] Serialized 6357 byte log entry
0506 15:14:48.608711 (+349774us) op_driver.cc:444] REPLICATION: finished
0506 15:14:48.884371 (+275660us) write_op.cc:301] APPLY: starting
0506 15:14:48.884477 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:48.903591 (+ 19114us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:48.920770 (+ 17179us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:48.953704 (+ 32934us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:48.953980 (+ 276us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0506 15:14:48.954030 (+ 50us) write_op.cc:312] APPLY: finished
0506 15:14:48.954381 (+ 351us) log.cc:842] Serialized 112 byte log entry
0506 15:14:48.954535 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:48.954717 (+ 182us) write_op.cc:454] Released schema lock
0506 15:14:48.954769 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:48.955039 (+ 270us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:48.955072 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":677095,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":270162,"apply.run_cpu_time_us":5032,"apply.run_wall_time_us":73386,"num_ops":16,"prepare.queue_time_us":441115,"prepare.run_cpu_time_us":6380,"prepare.run_wall_time_us":174622,"replication_time_us":408668,"thread_start_us":144128,"threads_started":1}]]}
I20240506 15:14:51.038415 20778 catalog_manager.cc:5484] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:51.040807 20778 catalog_manager.cc:5484] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:51.034773 20877 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:51.008402 22703 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.005707 22708 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4.
I20240506 15:14:51.048087 20998 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.054081 20778 catalog_manager.cc:5484] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:50.989388 22310 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.019675 22714 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.049474 20877 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240506 15:14:51.022672 22717 consensus_queue.cc:1046] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20240506 15:14:51.042685 22435 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.044765 22521 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:51.029928 22354 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
W20240506 15:14:51.064448 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 208828us
I20240506 15:14:51.063025 20778 catalog_manager.cc:5484] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:14:51.065740 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 188842us
I20240506 15:14:51.076066 21032 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:51.043480 22718 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20240506 15:14:51.061852 22530 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:51.046074 22706 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240506 15:14:51.092666 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.035902 22579 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.095345 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.109441 20878 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:51.096647 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:51.094184 20778 catalog_manager.cc:5484] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:14:51.110589 20877 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:51.050877 22634 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20240506 15:14:51.103053 20924 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.076975 21686 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:51.108147 20998 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.118032 20998 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.068827 22628 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.073644 22435 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:51.054807 22703 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.104115 22620 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20240506 15:14:51.103389 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.066501 22648 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.116701 22365 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.083910 22646 raft_consensus.cc:686] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:51.121219 22645 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:51.095496 22617 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.139900 20998 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.092816 22622 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:51.103250 22558 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.113945 22641 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.104530 22618 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.075186 22708 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20240506 15:14:51.121239 22581 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.104388 22636 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.189424 22636 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:51.189852 22711 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.384s user 0.003s sys 0.000s
I20240506 15:14:51.161350 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.191444 22636 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.162699 22617 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.192497 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.097414 22530 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.193885 22530 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:51.194348 22617 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.195690 22708 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:51.105958 22477 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.196642 22477 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.197904 22579 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:51.198618 22477 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.073208 22714 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.158303 22365 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.134478 22559 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:51.122534 22628 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.205332 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.172154 20877 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:51.104820 22310 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.168311 22558 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.162691 22500 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240506 15:14:51.176950 22703 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.177592 22624 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.133729 21656 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:51.177911 22354 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:51.166635 22661 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240506 15:14:51.257333 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:51.156467 22645 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.166827 22717 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.159919 22556 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.168092 22641 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.155769 22586 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.188807 22581 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.104225 22642 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.209439 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.137594 22718 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.321997 22718 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.322463 22718 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.213374 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.215865 20879 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:51.159855 22638 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.196111 22708 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.325491 22708 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:51.239660 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:51.187850 20949 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:51.326529 22484 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.249292 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.250125 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.328691 21027 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:51.328933 21030 raft_consensus.cc:2379] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:51.326809 21029 raft_consensus.cc:2362] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4. Re-sending same reply.
I20240506 15:14:51.329349 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.255302 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:51.256337 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:51.256520 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.256745 22365 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:51.256678 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 145294us
I20240506 15:14:51.216723 22628 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.204901 22559 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.265051 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:51.325191 22638 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.334693 22669 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:51.335403 22638 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.337253 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.345118 20956 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:51.319926 22642 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.350729 22642 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.351526 22559 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:51.366950 20953 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:51.370368 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.268029 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.141721 22648 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.374212 22648 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.277868 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.279165 22717 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.272204 22675 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:51.376057 22717 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.376324 22675 raft_consensus.cc:686] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:51.377521 22675 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.272048 22645 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:51.380928 20876 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:51.264729 22691 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:51.200978 22618 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.386001 22618 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.167474 22496 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:51.386775 22496 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.388893 22496 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:51.372114 20953 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:51.390103 20876 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:51.390544 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:51.395015 22718 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.399665 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:51.401351 22707 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.401739 22707 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.402101 22707 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.401898 22634 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240506 15:14:51.409291 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 293731us
I20240506 15:14:51.410265 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:51.411778 22688 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.412568 22688 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.413137 22496 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.415732 20924 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.417121 22496 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.417608 22496 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.418006 22496 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.418541 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:51.420416 22703 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.415406 22708 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.425230 21032 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:51.427582 22619 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:51.281347 22568 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.279449 22714 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.242049 22624 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.434922 22525 raft_consensus.cc:3032] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:51.435351 22525 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:51.436478 22624 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.314931 22586 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.437292 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.438820 22586 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.278604 22556 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.443926 20998 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.389046 22717 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.422410 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.286540 22641 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.460383 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.462417 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:51.464679 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.465543 20952 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:51.468567 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:51.468340 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.318794 22477 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:51.475211 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.470110 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.485486 20955 raft_consensus.cc:2354] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:51.456156 22556 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.498315 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:51.499224 20957 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:51.461588 22624 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.500686 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.517621 20957 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:51.501058 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.499209 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.515245 21028 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:51.396492 22628 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.467653 22310 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.415652 22688 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.465391 22662 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:51.436144 22525 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.525171 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.521093 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.545962 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.556435 20846 leader_election.cc:398] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:51.556794 20846 leader_election.cc:401] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:51.557943 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.467363 22618 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.550066 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.562358 20958 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:51.467552 22638 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:51.564960 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.578424 20954 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:51.580724 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.583812 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.586266 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:51.524366 22624 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.316380 22622 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.553536 22310 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:51.602982 20846 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.551410 22617 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.567189 22586 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:51.574651 22579 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.621214 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.273216 22646 consensus_queue.cc:237] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.597956 22714 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:51.593852 22525 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:51.575814 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.577422 22662 raft_consensus.cc:686] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:51.619256 22310 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.600376 22624 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:51.498212 22477 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.598543 22638 raft_consensus.cc:686] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:51.629992 22558 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.590538 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.604585 22733 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.575435 22618 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.632743 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.608505 22619 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.627470 22703 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:51.608172 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.671934 20956 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
W20240506 15:14:51.656262 22727 log.cc:924] Time spent T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.495s user 0.001s sys 0.000s
I20240506 15:14:51.644215 22744 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:51.682149 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.682240 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.682958 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.692230 20956 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:51.627404 22617 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:51.656615 20951 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:14:51.692082 22729 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.377s user 0.001s sys 0.000s
I20240506 15:14:51.700495 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:51.654235 22444 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.052s sys 0.073s
W20240506 15:14:51.697674 22737 log.cc:924] Time spent T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.418s user 0.000s sys 0.001s
I20240506 15:14:51.668704 22708 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.634539 22707 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.710731 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.666886 22559 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.703924 22703 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.682570 22574 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.718969 20956 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:51.720196 20957 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:51.692312 22718 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.696295 22739 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:51.728046 22444 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.052s sys 0.073s
I20240506 15:14:51.680999 22619 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.700296 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:51.733103 22734 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.280s user 0.000s sys 0.001s
I20240506 15:14:51.768188 22638 consensus_queue.cc:237] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.782347 22714 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.793219 22717 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:51.710971 22645 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.794605 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:51.702317 22744 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:51.807824 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 107658us
I20240506 15:14:51.846624 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:51.808291 22738 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.120s user 0.000s sys 0.002s
I20240506 15:14:51.806856 22628 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:51.848009 20846 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:51.828478 22703 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:51.774494 22574 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:51.897890 22675 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.048s sys 0.034s
W20240506 15:14:51.880390 20880 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.007s sys 0.023s
W20240506 15:14:51.861898 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 491874us
W20240506 15:14:51.925580 20880 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.008s sys 0.023s
I20240506 15:14:51.790180 22581 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.801472 22496 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.949002 20949 raft_consensus.cc:2396] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:51.759974 22662 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.912637 22586 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240506 15:14:51.930747 22734 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20240506 15:14:51.925356 22568 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:51.843111 22714 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:51.939254 22739 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:51.967285 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1056) took 1090 ms. Trace:
I20240506 15:14:51.943070 22703 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.944424 22569 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:51.958174 20951 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:51.971768 20883 rpcz_store.cc:276] 0506 15:14:50.876450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:50.878129 (+ 1679us) service_pool.cc:225] Handling call
0506 15:14:50.878436 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:51.044238 (+165802us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:51.045704 (+ 1466us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:51.046143 (+ 439us) log.cc:842] Serialized 26 byte log entry
0506 15:14:51.120096 (+ 73953us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:51.120243 (+ 147us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:51.198566 (+ 78323us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b54001473d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:51.198612 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:51.965122 (+766510us) raft_consensus.cc:1671] finished
0506 15:14:51.965144 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:51.967185 (+ 2041us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11547,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":161978112,"thread_start_us":135953,"threads_started":2,"wal-append.queue_time_us":569616}
W20240506 15:14:51.970510 22749 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
W20240506 15:14:51.968379 22748 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
I20240506 15:14:51.971347 22477 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.001282 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.959254 22586 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.004410 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.014927 20883 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:52.016216 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:51.996529 22746 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.305s user 0.000s sys 0.001s
I20240506 15:14:52.021131 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:51.996969 22617 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:52.021735 22752 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.301s user 0.001s sys 0.000s
I20240506 15:14:52.035437 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.003973 22618 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.018942 22586 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.039323 20883 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:52.053133 22751 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240506 15:14:52.055384 22714 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:52.038187 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 190263us
I20240506 15:14:51.987058 22739 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.017726 22703 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.020113 22707 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.072105 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:52.034123 22743 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20240506 15:14:52.056190 22714 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.067448 22739 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.080160 20950 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:52.018015 22559 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.032012 22619 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.070518 20883 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
W20240506 15:14:52.060962 22745 log.cc:924] Time spent T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
I20240506 15:14:52.082190 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.084165 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:52.069016 22530 consensus_meta.cc:287] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.875s user 0.004s sys 0.008s
W20240506 15:14:52.077364 22742 log.cc:924] Time spent T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20240506 15:14:52.091249 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.095860 20878 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.092289 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.097951 20882 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:52.082731 22645 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.099184 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.085422 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.085347 22714 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:52.100421 22756 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20240506 15:14:52.090215 22574 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.109109 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.096005 22496 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.075779 22703 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:52.125069 22753 log.cc:924] Time spent T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20240506 15:14:52.125810 22759 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
W20240506 15:14:52.125870 22521 consensus_meta.cc:287] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 1.047s user 0.009s sys 0.003s
I20240506 15:14:52.127522 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.128660 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.129173 20879 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 3
W20240506 15:14:52.131769 22435 consensus_meta.cc:287] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.995s user 0.008s sys 0.004s
W20240506 15:14:52.132716 22760 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20240506 15:14:52.134183 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.135280 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:52.125232 22758 log.cc:924] Time spent T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20240506 15:14:52.119602 22641 consensus_meta.cc:287] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.652s user 0.006s sys 0.004s
I20240506 15:14:52.163252 20879 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:52.105898 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:52.119444 22556 consensus_meta.cc:287] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.572s user 0.005s sys 0.006s
I20240506 15:14:52.093684 22530 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.153764 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.169792 21025 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:52.166194 21026 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:52.162179 20882 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:52.106601 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:52.169755 22530 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.150952 20883 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:52.155263 20878 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:52.165912 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.192540 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.171622 20876 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:52.191730 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.195497 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:52.198627 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60723.8us
W20240506 15:14:52.191792 22579 consensus_meta.cc:287] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.540s user 0.012s sys 0.000s
I20240506 15:14:52.166417 22556 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.206363 22661 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:52.206976 22661 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.209255 22661 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:52.200683 22762 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240506 15:14:52.215684 22757 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20240506 15:14:52.201035 22763 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.151919 22435 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.164695 22641 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.220470 20878 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:52.205905 22579 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:52.227908 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.228953 20882 raft_consensus.cc:2379] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 3: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:52.220335 22763 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.244997 20998 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.247841 20877 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
W20240506 15:14:52.235531 22624 consensus_meta.cc:287] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.563s user 0.009s sys 0.002s
I20240506 15:14:52.248786 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.250118 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.259271 22624 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.227347 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.261296 22624 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.261231 22435 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.252103 21032 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:52.261873 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:52.262760 22703 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.245200 22763 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.263778 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:52.263182 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.265632 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:52.262856 20953 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:52.267293 22556 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.267519 20951 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:52.268729 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:52.269217 22766 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240506 15:14:52.269408 20951 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
W20240506 15:14:52.271174 22754 log.cc:924] Time spent T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.346s user 0.000s sys 0.001s
W20240506 15:14:52.274680 22310 consensus_meta.cc:287] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.001s
I20240506 15:14:52.276397 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.277860 20951 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:52.279693 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:52.284813 22714 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.285193 20845 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:52.285315 22740 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.632s user 0.000s sys 0.001s
I20240506 15:14:52.286302 22641 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.287212 22709 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:52.287356 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.295616 20954 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:52.296602 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:52.297391 20954 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:52.297451 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 48751.4us
I20240506 15:14:52.298269 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.289033 20955 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:14:52.298879 22584 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:52.289180 20957 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:52.299757 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:52.298210 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:52.300601 20955 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:52.300999 20954 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:52.301786 20955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:52.302552 20955 raft_consensus.cc:595] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:14:52.271225 22525 consensus_meta.cc:287] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.617s user 0.011s sys 0.000s
I20240506 15:14:52.304740 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:52.311206 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1089) took 1044 ms. Trace:
I20240506 15:14:52.305477 20951 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:52.258807 22586 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.326385 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:52.295475 22435 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.327173 21032 raft_consensus.cc:2354] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:52.327538 20882 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:52.327857 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1080) took 1120 ms. Trace:
I20240506 15:14:52.328198 20875 rpcz_store.cc:276] 0506 15:14:51.207328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.310468 (+103140us) service_pool.cc:225] Handling call
0506 15:14:51.316309 (+ 5841us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:51.415983 (+ 99674us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:51.416018 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:14:51.416616 (+ 598us) log.cc:842] Serialized 52 byte log entry
0506 15:14:51.674365 (+257749us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:51.674404 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:51.674481 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:52.270467 (+595986us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:52.323296 (+ 52829us) raft_consensus.cc:1671] finished
0506 15:14:52.323316 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.327795 (+ 4479us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98921,"raft.queue_time_us":102917,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":6449024,"thread_start_us":257380,"threads_started":1,"wal-append.queue_time_us":389495}
I20240506 15:14:52.320583 20876 rpcz_store.cc:276] 0506 15:14:51.266849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.201543 (+934694us) service_pool.cc:225] Handling call
0506 15:14:52.201827 (+ 284us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:52.202287 (+ 460us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.202314 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:14:52.204747 (+ 2433us) log.cc:842] Serialized 52 byte log entry
0506 15:14:52.295960 (+ 91213us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:52.296374 (+ 414us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.310820 (+ 14446us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.310918 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:52.310972 (+ 54us) raft_consensus.cc:1671] finished
0506 15:14:52.310993 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.311145 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14132,"prepare.queue_time_us":1183,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":36,"raft.queue_time_us":80,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":7726,"thread_start_us":61940,"threads_started":1,"wal-append.queue_time_us":73999}
I20240506 15:14:52.312302 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1395) took 1056 ms. Trace:
I20240506 15:14:52.317248 22707 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.288498 22310 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.339130 20845 leader_election.cc:398] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:52.321915 22584 raft_consensus.cc:686] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:52.327021 22525 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:52.325346 22709 raft_consensus.cc:686] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:52.343205 20957 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:52.330346 22772 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.321794 21686 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:52.331426 22435 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.338300 22714 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.343415 20845 leader_election.cc:401] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240506 15:14:52.349011 20952 raft_consensus.cc:2354] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
W20240506 15:14:52.346529 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 48295.1us
I20240506 15:14:52.346195 20875 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:52.349614 20957 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:52.350804 22579 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.352365 22435 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.350834 22584 consensus_queue.cc:237] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.344156 22707 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.353761 22744 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.354636 22707 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.350858 22622 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:52.350162 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:52.362805 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:52.366575 20957 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:52.356639 22525 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:52.367334 22703 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:52.369406 22310 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.380513 22750 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20240506 15:14:52.380862 22586 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.355904 22772 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.382336 22763 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.357199 22714 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.359650 22622 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.384389 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:52.379671 22703 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.384045 22731 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:52.395586 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 53077.9us
I20240506 15:14:52.385895 22641 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.375381 22707 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
W20240506 15:14:52.401755 22755 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.405s user 0.002s sys 0.001s
I20240506 15:14:52.402034 22763 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.402642 22714 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.403040 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1081) took 1195 ms. Trace:
I20240506 15:14:52.403298 22525 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.403405 20881 rpcz_store.cc:276] 0506 15:14:51.207608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.321069 (+113461us) service_pool.cc:225] Handling call
0506 15:14:51.321469 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:51.421761 (+100292us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:51.421805 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:51.422269 (+ 464us) log.cc:842] Serialized 26 byte log entry
0506 15:14:51.804696 (+382427us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:51.804732 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:51.804797 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:52.399120 (+594323us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:52.402766 (+ 3646us) raft_consensus.cc:1671] finished
0506 15:14:52.402787 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.402985 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99872,"raft.queue_time_us":101201,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":90,"spinlock_wait_cycles":7080832,"thread_start_us":382128,"threads_started":1,"wal-append.queue_time_us":539509}
I20240506 15:14:52.395062 22703 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:52.410225 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:52.378810 22709 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.415923 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.417484 21026 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:52.417574 21032 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.419255 20845 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.430747 21032 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:52.433545 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.440457 22779 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:14:52.443538 22767 log.cc:924] Time spent T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.228s user 0.000s sys 0.002s
I20240506 15:14:52.432070 22714 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.445283 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1396) took 1195 ms. Trace:
I20240506 15:14:52.445508 22714 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.446136 22714 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.449959 22568 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.431324 22569 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.445762 21025 rpcz_store.cc:276] 0506 15:14:51.250324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.250664 (+ 340us) service_pool.cc:225] Handling call
0506 15:14:52.445078 (+1194414us) spinlock_profiling.cc:243] Waited 910 ms on lock 0x7b54001c1bd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.445222 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2092238080}
I20240506 15:14:52.398322 22731 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.406531 22641 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.450501 22568 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.438266 22444 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
W20240506 15:14:52.472828 22780 log.cc:924] Time spent T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20240506 15:14:52.472900 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.477871 21025 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:52.473775 21032 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:52.458173 22622 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.480736 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.480847 22444 raft_consensus.cc:686] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:52.487393 22586 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240506 15:14:52.488443 22444 consensus_queue.cc:237] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.341100 21023 rpcz_store.cc:276] 0506 15:14:51.250040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.256808 (+ 6768us) service_pool.cc:225] Handling call
0506 15:14:51.257140 (+ 332us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:51.317828 (+ 60688us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:51.397204 (+ 79376us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:51.551903 (+154699us) log.cc:842] Serialized 26 byte log entry
0506 15:14:51.961506 (+409603us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:52.119867 (+158361us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.194904 (+ 75037us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.202166 (+ 7262us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:52.288649 (+ 86483us) raft_consensus.cc:1671] finished
0506 15:14:52.291744 (+ 3095us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.306278 (+ 14534us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5657,"prepare.queue_time_us":150035,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":2059,"raft.queue_time_us":77586,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":87890,"thread_start_us":486976,"threads_started":2,"wal-append.queue_time_us":511667}
I20240506 15:14:52.492048 22772 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.496910 20952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35816 (request call id 909) took 1025 ms. Trace:
I20240506 15:14:52.497325 20952 rpcz_store.cc:276] 0506 15:14:51.471346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.474892 (+ 3546us) service_pool.cc:225] Handling call
0506 15:14:52.496693 (+1021801us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b540019e954. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.496844 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1986325376}
I20240506 15:14:52.465488 22569 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.514829 20952 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:52.500787 22619 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:52.497045 22628 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:52.542395 22788 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20240506 15:14:52.542703 22645 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.551100 22731 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.551823 22731 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:52.553416 22558 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:52.554440 20846 leader_election.cc:334] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545): Timed out: RequestConsensusVote RPC to 127.19.67.130:40545 timed out after 1.607s (SENT)
I20240506 15:14:52.554881 20846 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:52.555640 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:52.561969 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 50042.6us
I20240506 15:14:52.566884 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.556433 22628 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.539047 22761 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.538069 22617 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.607856 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 910) took 1136 ms. Trace:
W20240506 15:14:52.604794 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 49223.2us
I20240506 15:14:52.619712 20956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 92) took 1121 ms. Trace:
I20240506 15:14:52.541167 22619 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:52.620697 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:52.545441 22618 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.625903 21654 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:52.627316 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.631779 20879 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:14:52.631652 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 269512us
I20240506 15:14:52.628561 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.628368 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.619230 20958 rpcz_store.cc:276] 0506 15:14:51.471621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.763107 (+291486us) service_pool.cc:225] Handling call
0506 15:14:52.133871 (+370764us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b58000a2998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:14:52.134057 (+ 186us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.134254 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:14:52.134543 (+ 289us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.550329 (+415786us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.607522 (+ 57193us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:52.607574 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:52.607615 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.607681 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.607802 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":367464,"prepare.queue_time_us":2085,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":368602,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":116,"spinlock_wait_cycles":20594048,"thread_start_us":104644,"threads_started":1,"wal-append.queue_time_us":177905}
I20240506 15:14:52.628993 22586 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:52.629323 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.607275 22794 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:52.633845 20878 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:52.633869 20881 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 3: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:52.627712 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1059) took 1748 ms. Trace:
I20240506 15:14:52.599843 22691 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:52.634958 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.620477 20956 rpcz_store.cc:276] 0506 15:14:51.499718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.783168 (+283450us) service_pool.cc:225] Handling call
0506 15:14:51.783536 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:51.783788 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 7
0506 15:14:51.784118 (+ 330us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.495370 (+711252us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b18000b93e0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b33e2ea 00007f6314a2dfe5 00007f630b3c23d4 00007f630b3c1cc5 00007f630e2e7620 00007f630e2f97c6 00007f630e30a327 00007f630e30a2a8 00007f630e30a228
0506 15:14:52.619391 (+124021us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:52.619447 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:14:52.619489 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.619550 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.619659 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68811,"prepare.queue_time_us":1459,"prepare.run_cpu_time_us":1421,"prepare.run_wall_time_us":6243,"raft.queue_time_us":715925,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":8914,"spinlock_wait_cycles":12005760,"thread_start_us":123725,"threads_started":1}
I20240506 15:14:52.603622 22579 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:52.600550 22641 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.607378 22638 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:52.639113 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:52.640159 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.641788 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.647725 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.631609 22619 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:52.649184 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:52.659480 20924 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:52.543238 22645 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.651051 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.651680 22744 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:52.651965 22624 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.669827 20878 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:14:52.651935 22794 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.653038 22310 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:52.634845 22628 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.654024 22569 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.637172 22618 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.655086 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.601686 22435 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.604229 22731 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.637156 20880 rpcz_store.cc:276] 0506 15:14:50.879457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:50.882205 (+ 2748us) service_pool.cc:225] Handling call
0506 15:14:50.882505 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:51.043294 (+160789us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:14:51.213368 (+170074us) log.cc:842] Serialized 14 byte log entry
0506 15:14:51.940428 (+727060us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:51.940508 (+ 80us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:51.940590 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.627510 (+686920us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400180bd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:52.627553 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.627669 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16820,"prepare.queue_time_us":59383,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":62546,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5822,"spinlock_wait_cycles":369181440,"thread_start_us":867233,"threads_started":2,"wal-append.queue_time_us":808640}
W20240506 15:14:52.656535 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 91403.7us
I20240506 15:14:52.662365 20881 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.564939 22558 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.638334 22750 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:52.681417 21032 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:52.673250 22496 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.624140 22617 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.657341 22556 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:52.619791 22761 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.690820 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:52.676726 22646 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.697289 22617 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.693011 22496 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.656287 22703 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:52.699659 22790 log.cc:924] Time spent T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20240506 15:14:52.691694 22750 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.762434 22444 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.756531 22556 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:52.764034 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 73316.1us
I20240506 15:14:52.677860 22624 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.698568 22795 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:52.673980 22744 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:52.765764 22646 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:52.703300 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 67605.5us
I20240506 15:14:52.777426 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.778213 20880 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:52.780769 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:52.694916 22568 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.770432 22496 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.793380 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.789978 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.797973 20880 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:52.799226 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:52.774497 22750 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:52.774163 22444 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:52.809643 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1397) took 1545 ms. Trace:
I20240506 15:14:52.810027 21028 rpcz_store.cc:276] 0506 15:14:51.264651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.264795 (+ 144us) service_pool.cc:225] Handling call
0506 15:14:52.809497 (+1544702us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54001c20d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.809597 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":377159040}
I20240506 15:14:52.772406 22618 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.806253 20996 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:52.823199 22707 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:52.823803 22707 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.827811 22789 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:52.823199 22779 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:52.828451 22789 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.828809 22779 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:52.829188 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.829483 22779 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.830394 20951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1181) took 1120 ms. Trace:
I20240506 15:14:52.830911 20951 rpcz_store.cc:276] 0506 15:14:51.712156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.347773 (+635617us) service_pool.cc:225] Handling call
0506 15:14:52.377862 (+ 30089us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:52.552583 (+174721us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.552627 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:52.553212 (+ 585us) log.cc:842] Serialized 26 byte log entry
0506 15:14:52.597283 (+ 44071us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:52.597353 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.597472 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:52.830150 (+232678us) raft_consensus.cc:1671] finished
0506 15:14:52.830180 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:52.830335 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":159305,"raft.queue_time_us":159447,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":59,"spinlock_wait_cycles":14080,"thread_start_us":43626,"threads_started":1,"wal-append.queue_time_us":214725}
I20240506 15:14:52.833010 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.833282 22645 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.835740 22794 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.836406 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1164) took 1376 ms. Trace:
I20240506 15:14:52.836557 22795 raft_consensus.cc:686] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:52.837018 20953 rpcz_store.cc:276] 0506 15:14:51.460202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.460366 (+ 164us) service_pool.cc:225] Handling call
0506 15:14:51.467512 (+ 7146us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.266984 (+799472us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.836096 (+569112us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b54003084d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:52.836334 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828588288}
I20240506 15:14:52.837731 22795 consensus_queue.cc:237] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.838829 20949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35816 (request call id 920) took 1341 ms. Trace:
I20240506 15:14:52.839134 20950 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:52.839242 20949 rpcz_store.cc:276] 0506 15:14:51.497478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.761753 (+264275us) service_pool.cc:225] Handling call
0506 15:14:52.838600 (+1076847us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b540024ed54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.838771 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239550208}
W20240506 15:14:52.830500 22797 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240506 15:14:52.840981 20950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1174) took 1243 ms. Trace:
I20240506 15:14:52.847391 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.826356 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.774564 22722 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20240506 15:14:52.847077 20950 rpcz_store.cc:276] 0506 15:14:51.597293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.100011 (+502718us) service_pool.cc:225] Handling call
0506 15:14:52.838786 (+738775us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.839554 (+ 768us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.840801 (+ 1247us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b540024ed50. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eec59 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.840932 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1698310912}
I20240506 15:14:52.850473 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:52.851184 21024 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.851071 20996 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:52.851800 20881 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:52.816376 22444 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:52.853729 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.856101 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.857748 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:52.860808 20874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 1066) took 1952 ms (client timeout 2481 ms). Trace:
W20240506 15:14:52.861236 20874 rpcz_store.cc:269] 0506 15:14:50.907997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:50.999365 (+ 91368us) service_pool.cc:225] Handling call
0506 15:14:52.860611 (+1861246us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54000bc054. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:52.860725 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2504369920}
I20240506 15:14:52.836410 22794 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.840493 22556 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:52.861696 22558 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.675455 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:53.675923 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.641152 22793 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:53.677027 21026 raft_consensus.cc:2379] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:53.676867 22793 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.678896 22793 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.678812 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:53.466185 22574 raft_consensus.cc:979] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:14:53.680322 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.680652 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:53.680914 21031 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:53.681088 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.681658 22574 raft_consensus.cc:1067] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:14:52.911989 22568 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:52.901985 22731 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:53.684357 22557 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:53.684870 22557 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:53.000036 22761 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.686767 22557 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:52.959300 22435 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:53.132807 22744 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:53.257974 22747 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.451s user 0.010s sys 0.018s
I20240506 15:14:52.928886 22789 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:52.929585 22714 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.688045 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
W20240506 15:14:53.130699 22802 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
W20240506 15:14:52.964107 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 284894us
I20240506 15:14:53.744860 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:53.745522 20880 raft_consensus.cc:2379] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:53.688326 22794 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.018640 21687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59110 (request call id 189) took 1371 ms. Trace:
W20240506 15:14:53.740411 22747 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.942s user 0.020s sys 0.031s
I20240506 15:14:53.690003 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:52.948514 20879 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
W20240506 15:14:53.746131 22444 consensus_meta.cc:287] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.883s user 0.010s sys 0.001s
W20240506 15:14:53.517781 22619 consensus_meta.cc:287] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.839s user 0.004s sys 0.010s
W20240506 15:14:53.369295 22496 consensus_meta.cc:287] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.560s user 0.010s sys 0.000s
I20240506 15:14:53.691388 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:53.028959 22707 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:53.102311 22669 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:53.009145 22636 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:53.499089 22804 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.683s user 0.001s sys 0.000s
I20240506 15:14:53.698277 22761 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:52.953725 22783 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:53.705507 22733 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:53.674384 21024 consensus_meta.cc:287] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.819s user 0.035s sys 0.032s
W20240506 15:14:53.156111 20877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.007s sys 0.005s
I20240506 15:14:53.054664 20882 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:53.806664 21024 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:53.807164 20882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1124) took 1014 ms. Trace:
I20240506 15:14:52.861236 22624 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:53.264227 22559 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.014s sys 0.038s
I20240506 15:14:53.697919 22731 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:53.697824 22568 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:53.229872 22638 consensus_meta.cc:287] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.573s user 0.007s sys 0.003s
I20240506 15:14:53.692906 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:53.692737 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:53.694053 20876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 111) took 1434 ms. Trace:
W20240506 15:14:53.370715 20829 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.013s sys 0.005s
I20240506 15:14:52.961508 22581 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.714128 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.715947 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.839169 21026 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:53.683235 20996 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:53.719022 22779 consensus_meta.cc:287] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.889s user 0.014s sys 0.000s
W20240506 15:14:53.724900 21025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.010s sys 0.017s
W20240506 15:14:53.842303 21025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s user 0.010s sys 0.017s
I20240506 15:14:53.726521 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:52.856530 22641 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.883s
W20240506 15:14:53.254062 20921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.000s sys 0.016s
W20240506 15:14:53.901201 20921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.226s user 0.000s sys 0.026s
I20240506 15:14:53.674993 22646 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.755743 20924 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:52.920394 22703 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:53.303512 22620 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:53.740763 22435 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:53.751176 22709 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.033s sys 0.038s
W20240506 15:14:53.947554 22709 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.145s user 0.033s sys 0.039s
I20240506 15:14:53.755242 22744 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:53.948583 22744 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:53.948146 22435 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.762142 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:53.949487 22744 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.942617 22620 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:53.782888 20952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1196) took 1563 ms. Trace:
I20240506 15:14:53.952440 20952 rpcz_store.cc:276] 0506 15:14:52.219445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.536070 (+316625us) service_pool.cc:225] Handling call
0506 15:14:52.536439 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:52.608316 (+ 71877us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.608348 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:52.618503 (+ 10155us) log.cc:842] Serialized 26 byte log entry
0506 15:14:52.683131 (+ 64628us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:52.683173 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.683242 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:53.782610 (+1099368us) raft_consensus.cc:1671] finished
0506 15:14:53.782638 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.782798 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87350,"raft.queue_time_us":81918,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":3692,"thread_start_us":44090,"threads_started":1,"wal-append.queue_time_us":178510}
W20240506 15:14:53.806701 20877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.176s user 0.012s sys 0.014s
I20240506 15:14:53.807530 20882 rpcz_store.cc:276] 0506 15:14:52.792430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:53.053907 (+261477us) service_pool.cc:225] Handling call
0506 15:14:53.054290 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:53.806902 (+752612us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:53.807111 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:53.954001 22810 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:53.954207 22620 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:53.954975 22558 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:14:53.954862 22810 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.955750 22558 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:53.947377 22714 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.958171 22794 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.958701 22558 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:53.959628 22750 consensus_meta.cc:287] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 1.138s user 0.011s sys 0.001s
I20240506 15:14:53.960216 22750 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:53.961776 21684 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:53.962733 22750 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:53.965168 22811 log.cc:924] Time spent T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20240506 15:14:53.967072 22809 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240506 15:14:53.969039 22814 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20240506 15:14:53.810503 22761 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.973599 22761 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.861446 22581 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:53.974400 22581 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.976408 22581 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.843369 22568 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.978737 22568 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.837698 22731 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.981500 22731 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.816556 22733 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.984113 22733 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.704757 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 95) took 1057 ms. Trace:
I20240506 15:14:53.829860 20876 rpcz_store.cc:276] 0506 15:14:52.260074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.334750 (+ 74676us) service_pool.cc:225] Handling call
0506 15:14:52.335027 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:52.341686 (+ 6659us) raft_consensus.cc:1520] Early marking committed up to index 6
0506 15:14:52.346953 (+ 5267us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.436493 (+ 89540us) log.cc:842] Serialized 14 byte log entry
0506 15:14:52.437346 (+ 853us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:52.919232 (+481886us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b48000d0458. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6313197431 00007f631318e527 00007f631318e3ba 00007f631328256f 00007f6313282237 00007f631329673c 00007f63132966e8 00007f6313296668 00007f6313296623 00007f63132958cc 00007f63122189c6
0506 15:14:53.155336 (+236104us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b2c00031b98. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630ccd1f11 00007f630cce1542 00007f630e32afc1 00007f630e2ef7da 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782
0506 15:14:53.155747 (+ 411us) log.cc:842] Serialized 26 byte log entry
0506 15:14:53.693172 (+537425us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:14:53.693508 (+ 336us) log.cc:842] Serialized 14 byte log entry
0506 15:14:53.693729 (+ 221us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:53.693810 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:53.693873 (+ 63us) raft_consensus.cc:1671] finished
0506 15:14:53.693890 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.694002 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5527,"prepare.queue_time_us":73638,"prepare.run_cpu_time_us":174,"prepare.run_wall_time_us":2589,"raft.queue_time_us":12343,"raft.run_cpu_time_us":3403,"raft.run_wall_time_us":601487,"spinlock_wait_cycles":8353536,"thread_start_us":76054,"threads_started":1,"wal-append.queue_time_us":117727}
W20240506 15:14:53.813611 22559 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.114s user 0.037s sys 0.045s
W20240506 15:14:53.837700 20829 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.076s user 0.014s sys 0.007s
I20240506 15:14:53.762434 22619 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.988550 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 942) took 1489 ms. Trace:
I20240506 15:14:53.814148 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:53.761771 22444 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.989915 22619 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.762606 22496 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:53.990975 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49818 (request call id 112) took 1493 ms. Trace:
I20240506 15:14:53.991439 22444 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.992275 22496 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.991384 20877 rpcz_store.cc:276] 0506 15:14:52.497475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.497626 (+ 151us) service_pool.cc:225] Handling call
0506 15:14:52.497953 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:52.499714 (+ 1761us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.499749 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:52.537198 (+ 37449us) log.cc:842] Serialized 26 byte log entry
0506 15:14:53.954300 (+1417102us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:53.954338 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:53.954407 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:53.990776 (+ 36369us) raft_consensus.cc:1671] finished
0506 15:14:53.990798 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.990921 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1413,"raft.queue_time_us":1717,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":14475,"thread_start_us":1323750,"threads_started":1,"wal-append.queue_time_us":1186955}
I20240506 15:14:53.993276 20883 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:53.993800 20878 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:53.826421 22624 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:53.994518 22624 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:53.995381 20878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 1123) took 1203 ms (client timeout 1547 ms). Trace:
I20240506 15:14:53.997009 22624 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:53.989110 21023 rpcz_store.cc:276] 0506 15:14:52.498989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.500712 (+ 1723us) service_pool.cc:225] Handling call
0506 15:14:52.544137 (+ 43425us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:52.636169 (+ 92032us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.636216 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:52.684411 (+ 48195us) log.cc:842] Serialized 26 byte log entry
0506 15:14:52.782408 (+ 97997us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:52.788011 (+ 5603us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.807917 (+ 19906us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:53.988199 (+1180282us) raft_consensus.cc:1671] finished
0506 15:14:53.988242 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.988470 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33723,"raft.queue_time_us":55024,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":9566,"thread_start_us":81952,"threads_started":1,"wal-append.queue_time_us":122443}
I20240506 15:14:53.998754 20956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 94) took 1351 ms. Trace:
I20240506 15:14:53.908285 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:53.837411 22638 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:53.941054 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 257882us
W20240506 15:14:53.943048 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 187362us
I20240506 15:14:53.999310 20956 rpcz_store.cc:276] 0506 15:14:52.647084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.668036 (+ 20952us) service_pool.cc:225] Handling call
0506 15:14:52.668374 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.812020 (+143646us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.812050 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:52.812091 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:14:52.812144 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:53.998465 (+1186321us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400305f54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:53.998523 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.998680 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329876736}
I20240506 15:14:53.999833 20950 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.000840 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:54.001838 20924 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:53.869199 22779 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.003931 20880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 1121) took 1212 ms. Trace:
I20240506 15:14:54.004463 20880 rpcz_store.cc:276] 0506 15:14:52.791437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.798890 (+ 7453us) service_pool.cc:225] Handling call
0506 15:14:54.003686 (+1204796us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54000fc154. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:54.003854 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2174017920}
I20240506 15:14:54.005270 22779 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:54.006057 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.006683 20952 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.007283 22628 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:54.007803 22628 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:54.032783 22628 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:53.987061 20955 rpcz_store.cc:276] 0506 15:14:52.647389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.649011 (+ 1622us) service_pool.cc:225] Handling call
0506 15:14:52.649336 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.686910 (+ 37574us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.686941 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:52.686982 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:52.687010 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:53.704472 (+1017462us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b54002495d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:53.704533 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.704689 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85918976}
I20240506 15:14:54.040125 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.040977 20955 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.007870 20950 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:54.045322 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.046964 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.008162 20951 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:54.049559 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.050453 20951 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 1.
I20240506 15:14:54.009985 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.052695 20957 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.010159 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.057771 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.010660 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.059620 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.061082 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.061872 20954 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.000845 20996 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.013185 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.064010 20952 raft_consensus.cc:2354] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:54.013890 22638 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:54.014606 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.065023 20956 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:14:53.995885 20878 rpcz_store.cc:269] 0506 15:14:52.792059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.792724 (+ 665us) service_pool.cc:225] Handling call
0506 15:14:53.995122 (+1202398us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400266954. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:53.995301 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2750914432}
I20240506 15:14:54.016791 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.066761 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.067320 20949 raft_consensus.cc:2396] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20240506 15:14:53.825522 20875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.005s sys 0.020s
I20240506 15:14:53.774154 22669 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.009295 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:54.068086 20875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s user 0.006s sys 0.021s
I20240506 15:14:53.947340 22703 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.069115 20875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1115) took 1289 ms. Trace:
I20240506 15:14:54.069607 20875 rpcz_store.cc:276] 0506 15:14:52.779487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.779666 (+ 179us) service_pool.cc:225] Handling call
0506 15:14:52.808423 (+ 28757us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.808614 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:14:53.201686 (+393072us) log.cc:842] Serialized 14 byte log entry
0506 15:14:54.068680 (+866994us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:54.068749 (+ 69us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:54.068808 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:54.068895 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:54.069046 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":53308,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":1259256,"threads_started":2}
I20240506 15:14:53.753785 21687 rpcz_store.cc:276] 0506 15:14:51.640297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.963018 (+1322721us) service_pool.cc:225] Handling call
0506 15:14:52.963438 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:52.963668 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.963697 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:52.963738 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:14:52.963784 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:52.987826 (+ 24042us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:53.011793 (+ 23967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:14:53.807235 22636 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:54.082083 20883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50468 (request call id 1112) took 1305 ms (client timeout 1602 ms). Trace:
W20240506 15:14:54.092931 20883 rpcz_store.cc:269] 0506 15:14:52.777078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.777218 (+ 140us) service_pool.cc:225] Handling call
0506 15:14:54.081856 (+1304638us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000fda54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:54.082006 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2780729600}
W20240506 15:14:53.883771 22718 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s user 0.013s sys 0.017s
W20240506 15:14:54.132540 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 52987.1us
I20240506 15:14:54.151175 20874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1125) took 1358 ms. Trace:
I20240506 15:14:54.152345 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1400) took 1327 ms. Trace:
I20240506 15:14:54.080744 22638 raft_consensus.cc:686] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:54.152647 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 151828us
I20240506 15:14:54.162560 21025 rpcz_store.cc:276] 0506 15:14:52.824572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:52.828434 (+ 3862us) service_pool.cc:225] Handling call
0506 15:14:52.828771 (+ 337us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:14:52.849434 (+ 20663us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:52.849468 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:14:52.856030 (+ 6562us) log.cc:842] Serialized 52 byte log entry
0506 15:14:53.842773 (+986743us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:14:53.842822 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:54.135489 (+292667us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5400139554. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:14:54.135560 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:54.135631 (+ 71us) raft_consensus.cc:1671] finished
0506 15:14:54.135652 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:54.152276 (+ 16624us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6405,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":99343,"spinlock_wait_cycles":27821952,"thread_start_us":980091,"threads_started":1,"wal-append.queue_time_us":1122496}
I20240506 15:14:54.130740 22707 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:54.210311 21026 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:54.077663 22703 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.156221 20957 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:54.215287 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:14:54.171870 22718 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.898s user 0.013s sys 0.022s
I20240506 15:14:54.222702 20998 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.158864 22708 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 15:14:54.152371 22444 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.210719 22709 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:54.209553 22733 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:54.215747 22688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.067s sys 0.109s
W20240506 15:14:54.284662 22688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s user 0.067s sys 0.109s
I20240506 15:14:54.238149 22779 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:54.227519 22559 consensus_queue.cc:1046] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.068s
I20240506 15:14:54.244881 22709 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.253396 22733 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.337011 22824 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.337141 22733 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:54.323040 22779 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:54.354856 22636 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:54.355670 22820 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20240506 15:14:54.339354 22810 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.354044 22779 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:14:54.356869 22815 log.cc:924] Time spent T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20240506 15:14:54.320734 22638 consensus_queue.cc:237] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.347043 22669 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.356161 22709 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:54.375959 22818 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20240506 15:14:54.365943 22703 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.345764 22824 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.388998 22824 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:54.426525 20846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.000s sys 0.044s
W20240506 15:14:54.427444 20846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.000s sys 0.045s
W20240506 15:14:54.430815 20848 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.026s sys 0.016s
W20240506 15:14:54.431375 20848 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.026s sys 0.016s
I20240506 15:14:54.164348 20874 rpcz_store.cc:276] 0506 15:14:52.792776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:53.691203 (+898427us) service_pool.cc:225] Handling call
0506 15:14:53.691473 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:53.691690 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:14:53.691937 (+ 247us) log.cc:842] Serialized 14 byte log entry
0506 15:14:54.150804 (+458867us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:14:54.150868 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:54.150909 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:54.150984 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:54.151117 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3149,"prepare.queue_time_us":51744,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":2033,"raft.queue_time_us":11429,"thread_start_us":455319,"threads_started":1}
W20240506 15:14:54.440284 20997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.029s sys 0.030s
W20240506 15:14:54.441031 20997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.838s user 0.029s sys 0.031s
W20240506 15:14:54.442528 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 227125us
W20240506 15:14:54.440372 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.016s
W20240506 15:14:54.444912 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.000s sys 0.017s
I20240506 15:14:54.446112 20996 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:54.439160 22841 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:54.448835 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 240114us
I20240506 15:14:54.448781 22841 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.227007 20956 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:14:54.459460 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 442682us
I20240506 15:14:54.459697 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.461112 22841 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.461658 21684 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:54.462006 22841 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.463236 22841 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.226855 22435 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.465584 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.466521 22435 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.238523 22444 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.468434 22703 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:54.467290 22435 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.440097 20883 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240506 15:14:54.479907 22581 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:54.480683 22628 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.481094 22581 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.482746 20958 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:54.451303 20921 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.488628 22714 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.489982 22521 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.489625 22714 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.492604 22619 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.494227 22733 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:54.493844 22619 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.494989 22435 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.495229 22733 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.495730 22435 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.497414 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.498006 22444 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.499890 22828 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.512115 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.513046 22828 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.497939 22826 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:54.505350 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.515141 22826 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.507763 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.516804 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.517071 20952 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.517709 20950 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.517990 22828 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.519104 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.501315 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:14:54.523267 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 72248.6us
I20240506 15:14:54.523778 22574 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 15:14:54.502647 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.533684 20951 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:54.502765 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.535296 20949 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.535537 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.503754 22688 raft_consensus.cc:3032] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.536461 22688 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:54.538012 22688 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.486059 22824 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:54.504112 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:54.504994 22558 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.014s sys 0.016s
I20240506 15:14:54.506086 22830 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:54.542691 22824 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.543217 22830 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.544638 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.545711 22824 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.545876 22830 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:54.506578 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:54.546870 22826 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.547173 20874 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:54.506958 22829 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.548472 21024 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:54.548713 22829 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:54.508641 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64734.3us
I20240506 15:14:54.508982 22584 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:54.487187 22568 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:14:54.493844 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 258782us
I20240506 15:14:54.492684 22731 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:54.524647 22435 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.525453 21023 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:54.502425 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.526857 20880 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:54.504698 22703 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.527689 21025 raft_consensus.cc:1261] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240506 15:14:54.529302 22628 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:54.530390 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.530655 22733 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.531739 22581 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.557559 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.534317 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:54.542428 22558 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.015s sys 0.016s
I20240506 15:14:54.571494 22435 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.571555 21687 raft_consensus.cc:2396] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240506 15:14:54.573844 20956 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.576198 22559 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.576596 22841 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.577590 22557 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:54.540200 22688 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.566112 22731 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.566964 21031 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:54.567127 20953 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:14:54.567612 22703 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.568662 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.554313 22444 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.581338 22731 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:54.581298 22444 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.582438 22521 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:54.568997 22829 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.563374 20921 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.583782 22444 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.584270 22628 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:54.584528 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:54.584687 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 60472.3us
I20240506 15:14:54.585453 20949 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:54.587023 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.587491 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.568559 22619 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:54.569685 22568 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.588546 22844 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.589551 22557 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:54.589324 22844 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.590214 22568 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.554548 22584 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:54.590459 22557 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:54.590874 22584 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.591535 20998 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.595722 22662 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:54.561115 22828 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.596858 22828 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.597163 22750 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:54.599396 22828 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.601438 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.602348 20949 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:14:54.604590 22824 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:54.563537 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 364347us
I20240506 15:14:54.605623 22662 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:54.592561 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.610500 21654 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:54.610649 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.611095 21024 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.611378 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.612138 22750 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:14:54.605193 22824 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.612547 22750 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.612738 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.612982 22750 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:54.613557 20949 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:54.614199 22824 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.590014 22559 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:54.615216 22559 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.591703 22844 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.615566 22559 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:54.615861 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:54.593559 22714 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.616688 22559 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.618147 22559 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.618505 21654 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:54.619431 22824 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:54.611706 21687 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:54.619796 22824 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:54.620229 22824 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:54.608139 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:54.592599 22557 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.625047 22821 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:54.629009 22733 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:54.629855 22733 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.630605 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.631971 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.631997 22733 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.632455 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.635128 21024 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:54.634948 22838 consensus_queue.cc:1046] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:54.633208 20950 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:54.639026 20845 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.646312 21029 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:54.648665 22834 raft_consensus.cc:3032] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.649231 22834 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:54.649420 22733 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:54.650080 22733 raft_consensus.cc:686] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:54.650514 22834 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.651304 22733 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.652652 22834 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.637295 22750 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.659869 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.660604 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.657215 22628 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:54.660482 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.665426 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.666086 20950 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:54.666672 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.668056 21025 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:54.669267 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.629012 22584 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.671702 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.675657 21026 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:54.672363 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.676726 22584 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:54.673553 21024 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.674134 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:54.672518 20922 consensus_peers.cc:485] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74536b2ff8754c03a925c7b726433266. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:14:54.679936 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.680776 20875 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:54.681963 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.680894 20922 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:54.682873 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.683928 22558 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:54.684389 21023 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:54.688680 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.689584 21031 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:54.690244 22824 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.691955 20880 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240506 15:14:54.692104 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.692592 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.693022 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.693125 20883 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:54.693178 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.693634 20953 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:54.693714 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.693982 20874 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:54.694574 20878 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:54.696131 20875 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
W20240506 15:14:54.689010 20955 consensus_meta.cc:287] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.647s user 0.007s sys 0.011s
W20240506 15:14:54.682727 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 95386.5us
I20240506 15:14:54.699616 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.677736 22750 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.702399 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.702349 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.703474 21032 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:54.709806 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.711102 22750 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:54.702463 22688 consensus_queue.cc:1046] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20240506 15:14:54.711863 22688 raft_consensus.cc:3032] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.708804 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.712239 22688 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:54.709486 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.714283 22688 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.716856 22750 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:54.717382 22750 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:54.718075 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.719280 20953 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:14:54.719802 20878 raft_consensus.cc:2396] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240506 15:14:54.720955 21654 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:54.722008 22838 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:14:54.704478 20955 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
W20240506 15:14:54.724689 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 54184.8us
I20240506 15:14:54.732118 20776 catalog_manager.cc:5484] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:54.734671 21024 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:54.716001 22688 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.738354 22838 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:54.745507 22838 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.747455 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.748922 20776 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 6 to 7, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:54.759465 20778 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 175) took 10195 ms. Trace:
I20240506 15:14:54.760854 22779 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.781010 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.781913 20876 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:54.783598 22779 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:54.761117 20776 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 3 to 5, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:54.785573 20776 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:54.764701 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.790593 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.791388 21023 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:54.766700 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 195) took 6955 ms. Trace:
I20240506 15:14:54.768241 20875 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:54.768870 20950 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:14:54.770128 20874 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:54.792461 20780 rpcz_store.cc:276] 0506 15:14:47.811566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:47.811729 (+ 163us) service_pool.cc:225] Handling call
0506 15:14:54.766437 (+6954708us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18835,"reported_tablets":20}
I20240506 15:14:54.793807 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.771759 20880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50474
I20240506 15:14:54.795075 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.772569 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:54.773397 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44396.5us
I20240506 15:14:54.775592 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.776019 22662 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:54.776477 22821 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:54.793951 21654 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.770069 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.763056 22750 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.795570 20880 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:14:54.800540 20880 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:54.701988 22834 consensus_queue.cc:1035] T faf3a7304753470787006e0ba7cf8852 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20240506 15:14:54.801559 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.803342 22558 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.803596 20877 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:54.803725 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.765656 22824 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.806478 20776 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:54.813359 21026 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:54.818091 22824 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:14:54.819144 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.820716 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.819963 20776 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:54.821952 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.822016 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.823513 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:54.823580 20874 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:54.832871 20877 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:54.837376 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 99832.8us
I20240506 15:14:54.838176 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:54.838686 21024 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:54.823200 22750 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.832720 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:54.841511 20883 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:14:54.841300 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.842933 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.848127 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.850206 20875 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:54.851908 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:54.852723 20875 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:14:54.856120 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:54.856604 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:54.856873 20878 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:54.858234 20883 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:54.862521 21023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:14:54.863250 21023 raft_consensus.cc:595] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:54.865815 20874 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:54.867172 22779 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.865489 20845 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.869907 20845 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.870834 22842 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:54.871240 22842 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:54.871608 22842 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.872162 22844 consensus_queue.cc:1046] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:54.873287 22844 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:54.873654 22844 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:54.874655 22844 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.875927 22844 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.876025 20875 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:54.879106 22847 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:54.868377 22779 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:54.879626 22847 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:54.838976 22824 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:54.880708 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:54.880995 22847 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.882123 22558 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:14:54.881814 22824 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.882828 20921 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.885645 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.883818 22558 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.860487 20880 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:54.894708 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 100856us
I20240506 15:14:54.894476 20845 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.897357 21024 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:54.898491 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:54.837195 20776 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 5 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:54.906832 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:54.907804 22444 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.896176 22733 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240506 15:14:54.913475 20880 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:54.918979 20845 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.930600 20776 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 4 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:54.911572 22834 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:54.917815 22714 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:54.934880 20877 raft_consensus.cc:2396] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240506 15:14:54.940490 20875 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:14:54.944643 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 59111.4us
I20240506 15:14:54.880772 22779 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.952315 22557 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:54.953016 22557 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.955219 22557 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:54.947211 22444 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:54.956125 22444 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:54.956522 22444 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:54.764271 20778 rpcz_store.cc:276] 0506 15:14:44.564198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:44.564494 (+ 296us) service_pool.cc:225] Handling call
0506 15:14:54.759111 (+10194617us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:51.713512 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:51.714017 (+ 505us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:51.714055 (+ 38us) write_op.cc:435] Acquired schema lock
0506 15:14:51.714073 (+ 18us) tablet.cc:660] Decoding operations
0506 15:14:51.715001 (+ 928us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:51.715061 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:51.715086 (+ 25us) tablet.cc:683] Acquiring locks for 9 operations
0506 15:14:51.715751 (+ 665us) tablet.cc:699] Row locks acquired
0506 15:14:51.715774 (+ 23us) write_op.cc:260] PREPARE: finished
0506 15:14:51.715926 (+ 152us) write_op.cc:270] Start()
0506 15:14:51.716052 (+ 126us) write_op.cc:276] Timestamp: P: 1715008491715878 usec, L: 0
0506 15:14:51.716073 (+ 21us) op_driver.cc:331] REPLICATION: starting
0506 15:14:51.736329 (+ 20256us) log.cc:842] Serialized 3645 byte log entry
0506 15:14:52.862351 (+1126022us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630b3a716e 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f630df704eb
0506 15:14:54.378964 (+1516613us) op_driver.cc:444] REPLICATION: finished
0506 15:14:54.393437 (+ 14473us) write_op.cc:301] APPLY: starting
0506 15:14:54.393544 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:54.393833 (+ 289us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:54.393864 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:54.394876 (+ 1012us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:54.395161 (+ 285us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0506 15:14:54.395225 (+ 64us) write_op.cc:312] APPLY: finished
0506 15:14:54.395605 (+ 380us) log.cc:842] Serialized 70 byte log entry
0506 15:14:54.395790 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:54.395952 (+ 162us) write_op.cc:454] Released schema lock
0506 15:14:54.396014 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:54.396352 (+ 338us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:54.396397 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":273536,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":8162,"apply.run_cpu_time_us":3379,"apply.run_wall_time_us":3379,"num_ops":9,"prepare.queue_time_us":442910,"prepare.run_cpu_time_us":55178,"prepare.run_wall_time_us":2045638,"raft.queue_time_us":394540,"raft.run_cpu_time_us":5111,"raft.run_wall_time_us":27232,"replication_time_us":2662346,"spinlock_wait_cycles":8084224,"thread_start_us":2615212,"threads_started":4,"wal-append.queue_time_us":2047134}]]}
I20240506 15:14:54.962056 20846 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:54.963443 22828 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:54.964212 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.965301 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:54.965821 22779 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.966791 20875 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:54.968061 22733 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:54.968462 20880 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:54.968746 22619 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:54.969298 22619 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:54.970321 22619 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.946480 20921 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:54.942471 22714 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:54.968642 22733 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:54.980808 22568 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240506 15:14:54.979216 22714 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:54.982815 22558 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:54.983472 20874 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:54.984513 22733 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:54.924562 22842 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:54.947566 22834 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:55.018208 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.042177 22857 ts_tablet_manager.cc:902] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:55.067772 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 70042.5us
I20240506 15:14:55.069280 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:14:55.073561 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 166666us
I20240506 15:14:55.093533 22847 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.099279 22844 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240506 15:14:55.146359 22859 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20240506 15:14:55.152899 22842 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:55.158999 22444 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.077566 22834 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:55.189743 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 197869us
I20240506 15:14:55.205263 20955 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240506 15:14:55.205942 21687 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:14:55.206737 22628 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.207341 22628 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:55.209108 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 85502.6us
I20240506 15:14:55.209717 22628 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.211869 22824 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:55.213878 22810 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.214676 22810 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.210048 20921 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:14:55.216750 22850 log.cc:924] Time spent T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20240506 15:14:55.186892 22558 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:55.234625 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43720.4us
I20240506 15:14:55.142391 22847 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.245384 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.233058 22628 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 15:14:55.243690 22857 tablet_copy_client.cc:316] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.130:40545
I20240506 15:14:55.265813 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.270781 22810 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.271211 22444 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.257123 22558 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.273886 22558 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.274024 22714 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.277266 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.278558 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.279372 20880 raft_consensus.cc:2379] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:55.279210 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:55.285908 21026 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:14:55.286423 21026 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.287313 21026 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:55.288401 22628 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.288789 22810 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.289111 22628 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.291375 21026 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:55.276039 22714 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.317795 20846 leader_election.cc:398] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:55.319234 20874 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:55.319455 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.323330 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:55.330055 20878 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:55.322815 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:55.293696 22444 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.311195 22828 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.336566 22714 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240506 15:14:55.350940 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 219179us
I20240506 15:14:55.357723 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.353770 22714 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.339895 22444 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.367863 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.370440 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.370930 20882 raft_consensus.cc:2396] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240506 15:14:55.364161 22628 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.369995 22714 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:55.376132 22828 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:55.386848 21687 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
I20240506 15:14:55.375308 22847 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.397145 21026 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240506 15:14:55.408816 22559 consensus_queue.cc:1046] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:55.409917 22559 raft_consensus.cc:3032] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:55.410341 22559 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.411216 22559 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.412504 20882 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240506 15:14:55.413949 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 49047.5us
I20240506 15:14:55.426201 20878 raft_consensus.cc:2454] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:14:55.429956 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.431025 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.431556 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.431735 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.429822 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.434105 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.434723 21029 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.435308 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.427290 20876 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:55.436931 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.432564 21030 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:55.433813 21027 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.441867 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.446161 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.446800 21025 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:14:55.447582 20875 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:55.448379 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:55.450433 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 213617us
I20240506 15:14:55.449267 21031 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:55.426940 22834 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20240506 15:14:55.442817 22619 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.460738 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.465010 21032 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:55.459564 22559 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.470039 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.470510 21028 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.468735 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.470880 21031 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:55.471836 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.458359 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.473606 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.474627 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.475622 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.475961 20845 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.476392 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.477334 22568 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:55.477758 22568 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:55.478164 22568 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.477339 22733 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240506 15:14:55.480569 22870 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20240506 15:14:55.480747 22733 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.481020 21027 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:55.481699 22828 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:55.482095 22828 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:55.482475 22828 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:55.482226 21656 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.475574 22842 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:55.483814 22842 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.484311 22842 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.481150 22733 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.456496 22688 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20240506 15:14:55.486563 22688 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:55.486908 22688 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.487627 22844 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:55.489665 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.492329 20846 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.487725 22688 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.496311 21028 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.497624 21031 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:55.455052 22584 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20240506 15:14:55.472734 20968 tablet_copy_service.cc:140] P ecededcd854842d4867fda9fa63a4b66: Received BeginTabletCopySession request for tablet 74536b2ff8754c03a925c7b726433266 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:44378)
I20240506 15:14:55.469810 22557 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240506 15:14:55.463356 22619 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:55.510939 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 64579.2us
I20240506 15:14:55.511206 22856 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:55.511693 20950 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240506 15:14:55.512343 22824 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:14:55.493655 21023 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.515167 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.501672 22844 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240506 15:14:55.522110 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57220.4us
I20240506 15:14:55.508232 22444 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.508704 22874 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.510046 22847 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:55.521755 20968 tablet_copy_service.cc:161] P ecededcd854842d4867fda9fa63a4b66: Beginning new tablet copy session on tablet 74536b2ff8754c03a925c7b726433266 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:44378: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-74536b2ff8754c03a925c7b726433266
I20240506 15:14:55.522977 22584 consensus_queue.cc:1225] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240506 15:14:55.523630 22844 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.543234 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.545707 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:55.556558 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 80543.4us
I20240506 15:14:55.534416 22688 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.554807 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.557971 22662 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.558219 21028 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:14:55.558647 21028 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.559736 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.559409 21028 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:55.560997 21028 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:55.567947 21028 raft_consensus.cc:1261] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240506 15:14:55.545137 22844 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:55.532680 22847 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:55.529811 22856 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20240506 15:14:55.533690 22444 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:55.532830 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.587164 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.577894 22568 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.589910 22568 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.590171 22828 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.592175 22828 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.594235 22662 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.583303 22444 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.583039 22847 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.595532 22444 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:55.598956 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 69849.7us
I20240506 15:14:55.599882 20921 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.600651 22874 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.602810 22834 consensus_queue.cc:1035] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:14:55.603207 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.604151 22874 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.604534 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.605410 22874 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:55.606021 20921 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.606920 22874 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.607354 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.608139 22874 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:55.614308 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.614996 21025 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:55.625532 22669 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:55.633687 20845 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:55.633441 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 46618.2us
I20240506 15:14:55.616436 22876 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:55.619606 22568 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:55.620566 22828 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:55.646631 22880 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:55.647133 22880 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.647581 22880 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:55.647142 22779 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20240506 15:14:55.652856 22733 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.654955 22733 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.659493 22880 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.633044 20968 tablet_copy_source_session.cc:205] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:14:55.662629 22880 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.655493 22733 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20240506 15:14:55.678014 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 90930.4us
I20240506 15:14:55.688216 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:55.641619 22568 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.645995 22828 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:55.697468 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 58078.7us
I20240506 15:14:55.663408 22880 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20240506 15:14:55.698098 22568 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:55.703845 22884 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20240506 15:14:55.706141 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.707019 22733 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:55.706871 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.638231 22885 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.066s
I20240506 15:14:55.708277 22877 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:55.677979 22619 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.708379 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:55.698297 22828 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:55.708707 22733 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
W20240506 15:14:55.716604 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 83005.8us
I20240506 15:14:55.717896 22887 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:55.725881 22892 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:55.731213 22844 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.731868 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.732591 22889 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:55.732709 22892 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.735780 22688 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:55.728030 22714 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.737438 20998 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.737686 22568 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.738292 22688 raft_consensus.cc:2781] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:14:55.738940 22688 raft_consensus.cc:686] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.739077 22828 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.739676 22844 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.739892 22688 consensus_queue.cc:237] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.741117 22828 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.742110 22842 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.742916 22568 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.744036 21029 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:14:55.708475 22874 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:55.664162 22876 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.744626 20845 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.717207 22733 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:55.746448 22876 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.747619 20998 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.747349 22733 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.748443 22856 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:14:55.748795 22733 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.754616 21025 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:14:55.738277 22714 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.753371 22444 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.755470 22619 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:55.756270 22887 raft_consensus.cc:686] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:55.738189 22847 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:55.763476 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 45810.8us
I20240506 15:14:55.760090 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.764714 20845 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:55.767069 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 58819.2us
I20240506 15:14:55.744154 22842 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.799121 20845 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:55.775811 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 74780us
I20240506 15:14:55.763407 22889 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:55.757201 22444 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.767561 22876 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
W20240506 15:14:55.778338 22882 log.cc:924] Time spent T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.174s user 0.003s sys 0.000s
I20240506 15:14:55.768468 22828 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:55.759163 22847 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.826488 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.828344 21028 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:55.767549 22662 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:55.775920 22877 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:14:55.783047 22883 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
W20240506 15:14:55.824790 20924 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.576s (SENT) [suppressed 5 similar messages]
I20240506 15:14:55.776083 22568 raft_consensus.cc:2741] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240506 15:14:55.833038 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.833340 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.843488 21026 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:55.839675 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:55.775889 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 247470us
I20240506 15:14:55.830013 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:55.778553 22892 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
I20240506 15:14:55.834200 22874 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:55.844801 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.856981 20875 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:55.857234 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.858232 22828 raft_consensus.cc:686] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:55.858371 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:55.858820 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.859414 22828 consensus_queue.cc:237] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.849273 22842 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:55.856853 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.862342 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.757228 22887 consensus_queue.cc:237] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.842628 22662 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:55.859957 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.869063 21023 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:14:55.871647 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.872053 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74536b2ff8754c03a925c7b726433266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:14:55.876873 21026 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.877899 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.878717 21026 raft_consensus.cc:2396] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240506 15:14:55.879581 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.852110 22874 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:55.881141 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.870126 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.882264 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.898042 21031 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:55.899888 21030 raft_consensus.cc:1226] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:14:55.881842 22874 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:55.901093 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.899056 21028 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:55.865307 22842 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:55.908838 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.909590 20883 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:55.867322 22876 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.917935 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.920331 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.921466 20875 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:55.922411 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.924078 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:55.926240 21687 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240506 15:14:55.925926 22662 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.926651 21687 pending_rounds.cc:76] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Aborting all ops after (but not including) 5
I20240506 15:14:55.926923 20876 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:55.928161 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.928360 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.928994 20875 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:55.929154 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:55.929447 20876 raft_consensus.cc:2379] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:55.929934 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:55.931253 20875 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:55.931723 20882 raft_consensus.cc:1226] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:55.932489 20878 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240506 15:14:55.933573 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.936317 22568 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:55.920941 22557 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:55.942991 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.945283 22557 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.950500 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.926939 21687 pending_rounds.cc:97] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240506 15:14:55.955127 21032 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:55.957393 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.958182 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:55.935990 22876 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.949705 22893 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:55.964228 21025 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240506 15:14:55.962743 21026 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:55.967698 22842 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.968206 22557 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:55.967890 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:55.972744 22897 log.cc:924] Time spent T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20240506 15:14:55.973801 22628 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:55.966861 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:55.955600 21687 raft_consensus.cc:2866] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:55.962337 22887 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.985827 20845 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:55.969717 22880 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:55.965922 22568 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:55.935820 22889 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:55.988407 20955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:55.989522 22891 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:55.990711 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:55.988155 22905 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:55.993695 22876 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:55.994794 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.009528 21028 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:56.010144 21027 raft_consensus.cc:2379] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:55.991849 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:56.011067 20879 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
W20240506 15:14:55.998279 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 117658us
I20240506 15:14:56.002321 21025 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:56.003134 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.012151 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.005810 20955 raft_consensus.cc:595] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:56.013365 20955 raft_consensus.cc:603] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:14:55.970902 22893 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:56.013860 20924 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.019893 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:56.021389 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.022018 21026 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:56.022025 21032 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:56.024960 22877 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:56.033293 22877 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:56.033825 22877 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.034927 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.035429 20876 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240506 15:14:56.027444 20883 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:56.009130 22619 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:56.021229 22880 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:56.044641 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.043151 22876 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:56.044910 22880 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.045643 22876 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:56.046082 22876 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.049536 22896 consensus_queue.cc:1046] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:14:56.050799 22896 raft_consensus.cc:3032] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:56.051250 22896 raft_consensus.cc:729] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:56.052060 22874 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.059870 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.075492 21024 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:56.024017 22907 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:56.036993 22521 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20240506 15:14:56.057399 22619 raft_consensus.cc:686] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
W20240506 15:14:56.083467 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 72469.5us
I20240506 15:14:56.092113 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.093284 20950 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:56.079072 22907 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:56.104329 21032 raft_consensus.cc:1226] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 4. Current term is 5. Ops: []
W20240506 15:14:56.109290 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 64756.9us
I20240506 15:14:56.112450 22917 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:56.084787 22876 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.114225 20953 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:56.064841 22881 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20240506 15:14:56.020884 22905 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.080132 22688 consensus_queue.cc:1046] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240506 15:14:56.138403 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.055452 22896 consensus_queue.cc:260] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:56.088111 22821 consensus_queue.cc:1046] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20240506 15:14:56.143817 22619 consensus_queue.cc:237] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.144793 22905 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.144626 22915 consensus_queue.cc:1046] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:14:56.111192 22907 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.087816 22891 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240506 15:14:56.150269 20953 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:56.155455 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:56.073280 22893 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.093420 22888 consensus_queue.cc:1046] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20240506 15:14:56.097173 22871 consensus_queue.cc:1046] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20240506 15:14:56.174492 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.176421 22889 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:56.178177 22877 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.182051 22917 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:56.182617 22917 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:56.183085 22917 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.188800 22877 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.189678 22889 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.193408 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 69) took 2488 ms. Trace:
W20240506 15:14:56.209934 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 54411.3us
I20240506 15:14:56.210474 22669 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:56.193861 20776 rpcz_store.cc:276] 0506 15:14:53.704788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:53.704975 (+ 187us) service_pool.cc:225] Handling call
0506 15:14:54.011970 (+306995us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54000037d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e274d30 00007f630e2f817a 00007f6314249b1c 00007f6314343a8e 00007f6312657a44 00007f63126579d2 00007f6312657901 00007f631265787c 00007f6312656b32 00007f630c716bcc 00007f630c716006
0506 15:14:56.193318 (+2181348us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:55.313944 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:55.314622 (+ 678us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:55.314677 (+ 55us) write_op.cc:435] Acquired schema lock
0506 15:14:55.314693 (+ 16us) tablet.cc:660] Decoding operations
0506 15:14:55.315945 (+ 1252us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:55.316063 (+ 118us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:55.316096 (+ 33us) tablet.cc:683] Acquiring locks for 8 operations
0506 15:14:55.396099 (+ 80003us) tablet.cc:699] Row locks acquired
0506 15:14:55.396127 (+ 28us) write_op.cc:260] PREPARE: finished
0506 15:14:55.432317 (+ 36190us) write_op.cc:270] Start()
0506 15:14:55.432507 (+ 190us) write_op.cc:276] Timestamp: P: 1715008495425025 usec, L: 0
0506 15:14:55.440600 (+ 8093us) op_driver.cc:331] REPLICATION: starting
0506 15:14:55.478037 (+ 37437us) log.cc:842] Serialized 3217 byte log entry
0506 15:14:55.660502 (+182465us) op_driver.cc:444] REPLICATION: finished
0506 15:14:55.868814 (+208312us) write_op.cc:301] APPLY: starting
0506 15:14:55.868929 (+ 115us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:55.869179 (+ 250us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:55.869321 (+ 142us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:55.870345 (+ 1024us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:55.870671 (+ 326us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0506 15:14:55.870728 (+ 57us) write_op.cc:312] APPLY: finished
0506 15:14:55.871078 (+ 350us) log.cc:842] Serialized 64 byte log entry
0506 15:14:55.872066 (+ 988us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:55.872199 (+ 133us) write_op.cc:454] Released schema lock
0506 15:14:55.872290 (+ 91us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:55.872556 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:55.879233 (+ 6677us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40316,"reported_tablets":10,"spinlock_wait_cycles":118118784,"child_traces":[["op",{"apply.queue_time_us":172929,"apply.run_cpu_time_us":3481,"apply.run_wall_time_us":26646,"num_ops":8,"prepare.queue_time_us":264803,"prepare.run_cpu_time_us":6120,"prepare.run_wall_time_us":164405,"replication_time_us":218300,"thread_start_us":162197,"threads_started":2}]]}
I20240506 15:14:56.210811 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.210990 22669 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:56.211524 22559 raft_consensus.cc:3032] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:56.211668 22874 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:56.212013 22559 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.212985 22559 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.213306 20924 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:56.214126 22559 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.214462 22874 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:56.214985 22874 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:56.215431 22874 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.217074 22888 raft_consensus.cc:3032] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:56.217532 22888 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.230043 21032 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:56.231418 20949 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:56.232563 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:56.233302 20953 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.238695 22619 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:56.239447 22521 raft_consensus.cc:3032] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:56.239890 22521 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.241216 22821 raft_consensus.cc:3032] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:56.241638 22821 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.172098 22905 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20240506 15:14:56.240775 22521 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.243901 20924 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:56.245378 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 130001us
I20240506 15:14:56.246220 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:14:56.228559 22922 log.cc:924] Time spent T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20240506 15:14:56.253419 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:56.255874 22928 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:56.256348 22928 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:56.256705 22928 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.254817 22521 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.259683 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:56.269412 20953 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:56.251726 22688 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:56.252503 21023 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:56.280752 20954 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:56.281487 22889 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.282342 20956 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:56.283258 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.276845 22877 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:56.266423 22628 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:56.242430 22821 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.286682 21024 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:56.287513 20883 raft_consensus.cc:1226] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 1. Current term is 2. Ops: []
I20240506 15:14:56.289255 20956 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:56.295292 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:56.273687 22688 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:56.308495 20955 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.223636 22888 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.288692 22876 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.290617 22628 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:56.402735 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 285018us
I20240506 15:14:56.527164 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:56.362473 20954 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:56.374573 20953 raft_consensus.cc:1203] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
W20240506 15:14:56.528488 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 182063us
I20240506 15:14:56.529018 20953 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20240506 15:14:56.383291 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 123672us
I20240506 15:14:56.517459 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:56.530572 20955 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:56.344218 22821 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.419207 22905 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:56.413980 22928 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.465935 22917 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:56.342867 22688 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:56.516884 22931 log.cc:924] Time spent T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
W20240506 15:14:56.516846 22933 log.cc:924] Time spent T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240506 15:14:56.526405 22888 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.532323 22628 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.540014 22889 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:56.572607 20883 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:56.403582 22662 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:56.580296 20956 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
W20240506 15:14:56.580977 22925 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20240506 15:14:56.584760 22877 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:56.585454 20882 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:56.591001 22917 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:56.592898 22688 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.594643 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:14:56.597523 22874 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.598513 22889 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:56.598124 22907 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.564886 22905 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:56.601773 22557 raft_consensus.cc:3032] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:56.570807 22669 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.608889 20881 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240506 15:14:56.606600 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 61260.9us
I20240506 15:14:56.544307 22444 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240506 15:14:56.631316 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.637195 20874 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:14:56.635104 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:56.624619 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 86615.3us
I20240506 15:14:56.640525 20952 raft_consensus.cc:1226] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:56.639360 20957 raft_consensus.cc:2362] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5. Re-sending same reply.
I20240506 15:14:56.652575 22521 consensus_queue.cc:1046] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:56.653735 22521 raft_consensus.cc:3032] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Stepping down as leader of term 1
I20240506 15:14:56.654120 22521 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.654842 22521 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.655663 22521 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 1 FOLLOWER]: Advancing to term 2
I20240506 15:14:56.657212 20956 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:56.658142 20957 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:14:56.661468 20958 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:56.663849 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.665397 20954 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:56.612393 22662 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.622489 22557 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.679980 20957 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:56.688123 20958 raft_consensus.cc:3032] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:14:56.627561 22917 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20240506 15:14:56.689464 20958 raft_consensus.cc:729] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:56.693964 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.697165 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:56.710348 20958 consensus_queue.cc:260] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.715595 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:56.700598 20954 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:56.718091 20957 raft_consensus.cc:2362] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5. Re-sending same reply.
I20240506 15:14:56.718921 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:56.724871 20953 raft_consensus.cc:2362] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5. Re-sending same reply.
I20240506 15:14:56.728652 22917 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:56.729199 22917 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:56.729564 22917 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.732873 22928 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.682612 22934 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.155s
I20240506 15:14:56.719533 22947 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:56.719148 22557 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.739217 20958 raft_consensus.cc:3037] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:56.697091 22877 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20240506 15:14:56.751931 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:56.761273 22628 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.775458 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.777228 22628 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:56.779083 22940 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240506 15:14:56.689074 22889 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.783041 21025 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:14:56.784453 21029 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:56.784613 21026 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:56.786266 22662 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:56.787688 20958 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20240506 15:14:56.788159 20958 pending_rounds.cc:76] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 2
I20240506 15:14:56.788455 20958 pending_rounds.cc:97] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20240506 15:14:56.788821 20958 raft_consensus.cc:2866] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:56.789316 21029 raft_consensus.cc:1203] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:14:56.789569 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:56.789798 21029 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240506 15:14:56.790674 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:56.791790 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:56.792476 22688 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:56.793073 22892 raft_consensus.cc:979] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:14:56.740389 22905 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:56.795786 22943 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:56.796171 22943 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:56.796952 22943 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.798146 22943 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.801633 21024 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:56.814510 22688 raft_consensus.cc:1067] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:14:56.814916 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.815754 21025 raft_consensus.cc:2354] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:14:56.816768 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:56.820969 22905 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:56.821593 22905 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:56.823045 21027 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:56.753684 22896 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.824810 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:56.756736 20950 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:56.767083 20996 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:56.828517 22688 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:14:56.828743 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 61735us
I20240506 15:14:56.829279 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:56.830718 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:56.769426 21023 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:56.768189 22521 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:56.771082 22907 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.750771 22557 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:14:56.740355 22932 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20240506 15:14:56.773311 22874 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.824380 22905 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:56.830595 22688 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 15:14:56.837466 22521 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:56.839495 22857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:34002 (request call id 104) took 1976 ms. Trace:
I20240506 15:14:56.839776 21032 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240506 15:14:56.874176 22857 rpcz_store.cc:276] 0506 15:14:54.863362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:54.863521 (+ 159us) service_pool.cc:225] Handling call
0506 15:14:55.042635 (+179114us) ts_tablet_manager.cc:903] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
0506 15:14:55.870661 (+828026us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:14:56.693529 (+822868us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:56.724534 (+ 31005us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:14:56.838725 (+114191us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:14:56.839031 (+ 306us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:56.839361 (+ 330us) ts_tablet_manager.cc:958] Replica registered.
0506 15:14:56.839428 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":66012,"tablet-copy.queue_time_us":126148,"thread_start_us":193465,"threads_started":2}
I20240506 15:14:56.848690 21687 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:56.852834 21656 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.858954 20996 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:56.848695 20921 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:14:56.856644 20880 raft_consensus.cc:1261] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20240506 15:14:56.880142 20921 leader_election.cc:401] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:56.880873 22557 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:56.881333 22947 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:14:56.862599 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:56.882992 20921 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.883931 22947 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:14:56.881367 22557 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:56.885265 22557 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.888689 22888 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:56.894522 22821 consensus_queue.cc:1046] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:56.863040 22917 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.863688 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:56.866901 20780 catalog_manager.cc:5484] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.895879 20879 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.896178 22662 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:56.752508 20957 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:56.896750 22662 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:56.844204 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 48074.3us
I20240506 15:14:56.857287 22521 consensus_queue.cc:1046] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:56.897940 22662 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.898661 22559 raft_consensus.cc:3032] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:56.898713 22938 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:56.899217 22559 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.899502 22688 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:56.900053 22559 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.879918 22905 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:14:56.902172 22688 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:14:56.902602 22905 consensus_queue.cc:1225] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240506 15:14:56.902755 22688 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:56.903312 22943 raft_consensus.cc:3032] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:56.903702 22943 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:56.903990 22688 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:56.904923 22943 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:56.935441 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 52501.3us
I20240506 15:14:56.896661 20780 catalog_manager.cc:5484] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.938234 20780 catalog_manager.cc:5484] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.940737 22889 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:56.943447 20780 catalog_manager.cc:5484] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:56.954427 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:56.952666 22688 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:14:56.958685 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 261592us
I20240506 15:14:56.959314 20996 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:56.961953 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.956871 20780 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 4 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.964854 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.953634 22559 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:56.965124 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.970022 20956 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:56.971087 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.971772 20957 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:56.961429 22943 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.960896 22938 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:56.958484 22857 tablet_copy_client.cc:791] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:14:56.973927 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.974078 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.977972 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.953155 22847 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240506 15:14:56.970419 22888 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:56.990576 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:56.994500 22917 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:56.994336 20921 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:56.995549 20879 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:56.997009 22907 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:56.975654 22821 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:56.999578 22821 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.000592 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:57.001924 22905 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:57.002030 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:57.002281 22821 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:56.973785 20780 catalog_manager.cc:5484] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.952651 22891 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20240506 15:14:57.002565 22905 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:56.958613 22842 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240506 15:14:56.982359 22947 consensus_queue.cc:1046] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:56.983647 22662 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:56.984357 22857 tablet_copy_client.cc:655] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:14:57.003474 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:57.007552 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.003753 20780 catalog_manager.cc:5484] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 4 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:56.967186 22444 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20240506 15:14:56.970588 22881 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
W20240506 15:14:57.009157 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 48012.5us
I20240506 15:14:57.014165 20780 catalog_manager.cc:5484] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.010762 22874 consensus_queue.cc:1046] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240506 15:14:57.020678 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.021417 20956 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.012753 22557 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.024127 22874 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:57.024318 22557 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:57.009780 22628 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:57.024152 21028 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:57.024894 22954 raft_consensus.cc:3032] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:57.024631 22874 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:57.003435 22915 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:57.028702 22954 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:57.032145 20876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:14:57.032809 20876 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:14:57.032855 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.033267 20876 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:57.030413 20921 leader_election.cc:398] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:14:57.011301 22947 raft_consensus.cc:3032] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:57.037531 20921 leader_election.cc:401] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:14:57.039265 20921 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:57.017277 22888 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.036659 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.030663 20780 catalog_manager.cc:5484] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.048388 22876 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:57.051555 22876 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.048449 22628 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.053560 20998 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:57.048843 21028 raft_consensus.cc:1226] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:57.057129 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.058121 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.059444 20879 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:57.059564 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.060933 21029 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:14:57.061767 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.061803 20780 catalog_manager.cc:5484] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.062999 22926 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20240506 15:14:57.032897 22954 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.065344 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.065770 22954 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:57.066196 20958 raft_consensus.cc:2396] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240506 15:14:57.066661 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.067267 20780 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:57.068099 21029 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:57.068758 20780 catalog_manager.cc:5484] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.069545 22929 consensus_queue.cc:1046] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:57.070665 22929 consensus_queue.cc:1046] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:57.070370 20780 catalog_manager.cc:5484] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.070973 22938 consensus_queue.cc:1035] T c89eb38afe3b43719809589e0aeb7eef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:57.071659 22929 raft_consensus.cc:3032] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:57.009385 22917 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.036s
I20240506 15:14:57.056814 22907 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:57.029054 22874 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:57.082415 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.083220 20998 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:57.072779 22928 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:57.084873 22928 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:57.085197 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.085696 22928 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.086822 22821 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.087396 22821 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:57.073127 22952 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240506 15:14:57.087752 22821 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.072368 22938 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240506 15:14:57.090276 22876 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.044863 22947 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:57.091045 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.045222 20876 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.071938 20780 catalog_manager.cc:5484] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.103363 20953 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:14:57.104626 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.106694 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.107570 20949 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:57.108950 20958 raft_consensus.cc:2379] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.113036 20876 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.114245 20780 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.117549 20780 catalog_manager.cc:5484] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:14:57.072455 22941 raft_consensus.cc:3032] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:57.080602 22871 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:57.125447 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.140532 22941 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.056532 22877 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.068s
I20240506 15:14:57.150218 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.150876 20954 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:57.141654 20780 catalog_manager.cc:5484] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.151803 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.072041 22929 raft_consensus.cc:729] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.137961 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.073611 22907 raft_consensus.cc:686] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:57.125759 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.086860 22928 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.144483 22619 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:57.146206 22880 consensus_queue.cc:1046] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:57.148983 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.153438 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.161785 20954 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.154793 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.155752 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.165154 20949 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.164171 21027 raft_consensus.cc:3032] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:57.171443 21027 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.153463 22957 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.152644 22871 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.177057 20780 catalog_manager.cc:5484] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.181737 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.190691 20955 raft_consensus.cc:2379] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.192981 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.194206 20958 raft_consensus.cc:2379] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.200126 20780 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.202812 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.221446 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:57.175730 22905 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:57.225354 20780 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.230734 22943 consensus_queue.cc:1046] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:57.231669 22943 raft_consensus.cc:3032] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:14:57.232023 22943 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:57.232491 22907 consensus_queue.cc:237] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.232805 22943 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.234351 22943 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:57.195880 22957 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:57.237421 22960 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.096s user 0.000s sys 0.003s
I20240506 15:14:57.238371 22887 raft_consensus.cc:3032] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:57.254895 22887 raft_consensus.cc:729] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:57.255339 22957 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.255718 22887 consensus_queue.cc:260] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.257344 22887 raft_consensus.cc:3037] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:57.257627 22967 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240506 15:14:57.237442 22917 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240506 15:14:57.268246 22963 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240506 15:14:57.240362 22929 consensus_queue.cc:260] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.282145 22929 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.242661 20958 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:57.243342 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.292187 20955 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.247715 20954 raft_consensus.cc:2454] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:57.252912 22628 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:14:57.253485 22619 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:57.294893 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80369.4us
W20240506 15:14:57.294989 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 75535.5us
I20240506 15:14:57.299635 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.313657 20780 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.315434 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.240976 22891 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.317828 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:57.299878 22961 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240506 15:14:57.321027 20996 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:57.289693 22915 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.323179 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.184031 22888 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.334233 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.340751 20954 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:57.316545 20780 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 5 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.288822 22876 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240506 15:14:57.168396 22947 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.342847 20996 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:57.339382 20950 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:57.342240 22972 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:57.299042 22619 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.353133 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.349202 20780 catalog_manager.cc:5484] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:14:57.347656 21027 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.174683 22941 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.322438 22891 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.344518 22896 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.391333 20954 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.402343 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:14:57.403071 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 60299us
I20240506 15:14:57.403165 21027 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:57.417940 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.419118 20955 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.396788 22557 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.346174 22915 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:57.434728 20956 raft_consensus.cc:2396] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:14:57.326234 22871 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:57.441821 22965 log.cc:924] Time spent T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240506 15:14:57.443011 22972 consensus_queue.cc:1046] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:14:57.447472 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.452312 20954 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.401298 22896 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:57.447512 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.423594 22557 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:57.454731 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.469718 21032 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.457162 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.461957 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.440910 22947 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:57.473999 21032 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240506 15:14:57.477551 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 71566.7us
I20240506 15:14:57.478489 20996 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:57.450191 22876 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.469725 22939 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:57.473052 22557 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.486090 22939 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.487125 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.487365 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.488198 20954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:57.488330 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.488752 20954 raft_consensus.cc:595] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:57.489082 20954 raft_consensus.cc:603] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:14:57.489713 20950 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:14:57.493374 21027 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:57.498548 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.504791 20845 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:57.506319 20845 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:57.507083 22847 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.507500 22847 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:57.507771 22957 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:14:57.508219 22847 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.509589 22889 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:57.510102 22889 raft_consensus.cc:686] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:57.511034 22889 consensus_queue.cc:237] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.514626 22921 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.515811 22892 raft_consensus.cc:3032] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:57.528867 20955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:14:57.529917 20955 raft_consensus.cc:595] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:14:57.530465 20955 raft_consensus.cc:603] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240506 15:14:57.533284 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 147095us
I20240506 15:14:57.518244 22892 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.519166 22972 raft_consensus.cc:2781] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:57.543311 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.543958 20879 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:57.547763 20958 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
W20240506 15:14:57.536634 22662 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.523s user 0.005s sys 0.006s
I20240506 15:14:57.515205 22921 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.501636 22876 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:57.556130 22921 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:57.557075 22971 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.557166 22876 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.541481 22943 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.557756 22971 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:57.568181 20952 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:14:57.569934 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.570072 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.578334 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:57.579392 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 75038.5us
I20240506 15:14:57.585932 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.587167 20879 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:57.588641 20949 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.591653 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.593585 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.594664 20876 raft_consensus.cc:2354] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:14:57.558669 22619 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.539888 22891 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.601270 20880 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:57.603225 20924 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:57.572501 22928 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.606951 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.607590 21026 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.607710 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.602895 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.605526 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.609200 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.608475 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.611864 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:57.612778 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.614174 20875 raft_consensus.cc:2379] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:57.558408 22943 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.615581 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:57.618749 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.587628 22972 raft_consensus.cc:686] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:57.617810 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.618579 22628 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:57.639854 22887 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.562860 22874 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:57.683248 22970 log.cc:924] Time spent T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20240506 15:14:57.587955 22821 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:57.706949 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 115224us
I20240506 15:14:57.640403 22954 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:57.609355 22892 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.647488 22971 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:57.642971 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:57.643605 22917 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:57.602028 20951 raft_consensus.cc:1226] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240506 15:14:57.706094 22896 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.645443 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.704694 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:57.620484 22915 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:57.707382 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 95629.4us
I20240506 15:14:57.728407 20876 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:57.646670 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.615739 22928 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.688387 22905 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:57.714141 22972 consensus_queue.cc:237] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.716665 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.732182 20954 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.717952 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:14:57.720805 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 77915.6us
I20240506 15:14:57.736382 20958 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:14:57.737154 22857 tablet_copy_client.cc:531] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:14:57.738929 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.739770 20924 leader_election.cc:398] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:14:57.739696 20958 raft_consensus.cc:2362] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5. Re-sending same reply.
I20240506 15:14:57.740829 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.704170 22887 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:57.722471 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 184441us
I20240506 15:14:57.723729 22929 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:57.724548 22876 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:57.638258 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:14:57.742159 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:57.721962 22628 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:57.731876 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:57.753357 22943 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.713089 22954 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:57.760757 20874 raft_consensus.cc:1203] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Deduplicated request from leader. Original: 2.1->[4.2-6.3] Dedup: 4.2->[6.3-6.3]
I20240506 15:14:57.761792 20954 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240506 15:14:57.762101 20954 pending_rounds.cc:76] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 2
I20240506 15:14:57.746650 22887 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.763594 20874 raft_consensus.cc:1226] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20240506 15:14:57.762292 20954 pending_rounds.cc:97] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.3
I20240506 15:14:57.766491 20954 raft_consensus.cc:2894] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Aborting config change with OpId 3.3: Aborted: Op aborted by new leader
I20240506 15:14:57.764844 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.767285 20958 raft_consensus.cc:2362] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5. Re-sending same reply.
I20240506 15:14:57.769166 20954 raft_consensus.cc:2866] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:57.772387 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.752077 22874 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.760447 22877 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:14:57.741171 22905 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.781461 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:57.784153 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.785213 20958 raft_consensus.cc:2379] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 4: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:14:57.765854 22821 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:57.750257 22929 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.788506 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.754390 22876 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.823568 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.824955 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.826124 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:57.827261 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.831846 22857 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:14:57.829205 20953 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:14:57.831506 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:14:57.832973 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:57.836316 20955 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:14:57.832136 20958 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:57.794507 20951 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:14:57.846732 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 254451us
I20240506 15:14:57.849126 21026 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:14:57.855809 20958 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20240506 15:14:57.856443 20958 pending_rounds.cc:76] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 3
I20240506 15:14:57.856492 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:57.857017 20958 pending_rounds.cc:97] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240506 15:14:57.857685 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.858055 20958 raft_consensus.cc:2866] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:57.858359 21025 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:57.859905 20955 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
W20240506 15:14:57.845371 22907 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.000s sys 0.019s
I20240506 15:14:57.854727 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44376 (request call id 3) took 6662 ms. Trace:
I20240506 15:14:57.869208 21653 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:57.869156 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:14:57.861133 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:14:57.861555 22907 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.001s sys 0.019s
I20240506 15:14:57.863680 20954 rpcz_store.cc:276] 0506 15:14:51.182788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:51.185771 (+ 2983us) service_pool.cc:225] Handling call
0506 15:14:51.331801 (+146030us) log.cc:842] Serialized 506 byte log entry
0506 15:14:52.415726 (+1083925us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b5400044820. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e288666 00007f630e281ffa 00007f630e281264 00007f630e269795 00007f630e26e9b1 00007f630e26e939 00007f630e26e8c9 00007f630e26e891 00007f630e26da5d 00007f6314639da4
0506 15:14:52.859848 (+444122us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b5400044820. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e280c9f 00007f630e269795 00007f630e26e9b1 00007f630e26e939 00007f630e26e8c9 00007f630e26e891 00007f630e26da5d 00007f6314639da4 00007f6314639bd9 00007f630b3c5236
0506 15:14:53.154678 (+294830us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b5400044820. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e280c9f 00007f630e269795 00007f630e26e9b1 00007f630e26e939 00007f630e26e8c9 00007f630e26e891 00007f630e26da5d 00007f6314639da4 00007f6314639bd9 00007f630b3c5236
0506 15:14:57.819796 (+4665118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119883,"raft.queue_time_us":26333,"raft.run_cpu_time_us":11679,"raft.run_wall_time_us":3333508,"spinlock_wait_cycles":116527616,"thread_start_us":390490,"threads_started":1,"wal-append.queue_time_us":580200,"wal-append.run_cpu_time_us":3148,"wal-append.run_wall_time_us":1135173}
I20240506 15:14:57.888428 22905 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.889539 22557 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.891127 22928 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:57.894922 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 119957us
I20240506 15:14:57.897399 20951 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:57.874065 22886 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.897929 22886 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:57.898260 22886 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.866513 22628 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:57.906256 20996 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:57.904538 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.907359 21032 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
W20240506 15:14:57.915769 20828 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20240506 15:14:57.938781 20921 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:57.939321 22966 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.762s user 0.005s sys 0.000s
I20240506 15:14:57.940189 20954 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:57.941387 21031 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:57.943874 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:57.945070 22993 log.cc:924] Time spent T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240506 15:14:57.947301 21656 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:57.948376 22979 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:57.948757 22979 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:57.949061 22979 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:57.950608 22954 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:57.954161 22939 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:57.947330 20955 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:57.947599 20950 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240506 15:14:57.975836 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 69674.2us
W20240506 15:14:57.976614 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 66383.1us
I20240506 15:14:57.959940 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:57.981516 20954 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240506 15:14:57.985327 20954 pending_rounds.cc:76] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 1
I20240506 15:14:57.985492 20950 raft_consensus.cc:1226] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:57.985707 20954 pending_rounds.cc:97] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240506 15:14:57.957479 20953 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:57.986129 20954 raft_consensus.cc:2866] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:14:57.892140 22557 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:57.987231 22874 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:14:57.984151 22847 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:57.991245 22874 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:57.991722 22874 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:57.992925 22847 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.993630 20880 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20240506 15:14:57.994488 22969 log.cc:924] Time spent T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.679s user 0.003s sys 0.000s
I20240506 15:14:57.994586 22876 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.996687 22983 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:57.997128 22983 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:57.997457 22983 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:57.997732 22917 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:14:58.004320 22954 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.004931 22954 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.006980 22954 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.007951 20951 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:14:58.009397 22887 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.011603 22887 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:57.981014 20956 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240506 15:14:57.957746 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 403215us
W20240506 15:14:58.025491 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 283373us
I20240506 15:14:58.027321 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.028954 20956 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.031445 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.036525 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:57.955499 22628 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:58.042820 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.043696 20950 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.045221 20924 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:58.046891 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:58.053776 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.055625 20958 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:58.056370 20924 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:58.053826 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.025120 22557 consensus_queue.cc:1046] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20240506 15:14:58.025537 22972 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.020906 22928 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:58.033993 22954 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.036011 22559 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.059549 22905 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:14:58.061060 22917 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:14:58.139775 20950 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.079826 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:58.089471 22929 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:58.207978 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 146432us
I20240506 15:14:58.094218 22877 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:58.221817 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 159704us
I20240506 15:14:58.080399 22954 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.081629 22928 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:58.084162 22559 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.073244 22972 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.221192 23000 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20240506 15:14:58.245286 23011 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20240506 15:14:58.246372 23012 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:58.250907 20922 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:58.275425 23012 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:14:58.276233 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:58.278195 22983 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.256856 23014 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
I20240506 15:14:58.257520 22988 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.288074 22988 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.220412 23002 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20240506 15:14:58.290426 22988 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:58.261420 22964 log.cc:924] Time spent T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.857s user 0.003s sys 0.000s
I20240506 15:14:58.262007 22874 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.292089 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1028) took 1272 ms. Trace:
I20240506 15:14:58.292428 22874 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.264616 22886 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:58.293854 21686 raft_consensus.cc:2379] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:14:58.292518 21023 rpcz_store.cc:276] 0506 15:14:57.019167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:57.020206 (+ 1039us) service_pool.cc:225] Handling call
0506 15:14:57.033332 (+ 13126us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:57.073791 (+ 40459us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:57.073832 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:57.074280 (+ 448us) log.cc:842] Serialized 26 byte log entry
0506 15:14:57.192273 (+117993us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:57.199036 (+ 6763us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:57.212946 (+ 13910us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:58.061277 (+848331us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:58.291727 (+230450us) raft_consensus.cc:1671] finished
0506 15:14:58.291755 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:58.292007 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40077,"raft.queue_time_us":40183,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":8843008,"thread_start_us":101862,"threads_started":1,"wal-append.queue_time_us":251325}
I20240506 15:14:58.264175 22972 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.268071 22928 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:14:58.269690 23017 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20240506 15:14:58.271121 22954 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.299346 22886 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.266142 22559 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:58.272305 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 527787us
I20240506 15:14:58.302865 22847 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.303432 22847 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.303810 22880 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.304644 22880 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.305552 22847 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.306124 22847 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:14:58.256623 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 74536b2ff8754c03a925c7b726433266, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.271214 20956 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.223383 22628 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.280742 22979 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:58.339008 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.340513 22954 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.357496 20882 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:58.357583 22954 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:58.358655 22979 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.359757 22954 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:58.341691 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.306989 22847 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.360540 20951 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:58.342162 21030 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
W20240506 15:14:58.343307 22962 log.cc:924] Time spent T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 1.084s user 0.002s sys 0.007s
I20240506 15:14:58.349472 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.362566 22847 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.349913 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.350667 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:58.363420 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:58.363902 20957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1252) took 1301 ms. Trace:
I20240506 15:14:58.351670 21032 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:14:58.352280 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:58.364305 20957 rpcz_store.cc:276] 0506 15:14:57.062348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:57.062454 (+ 106us) service_pool.cc:225] Handling call
0506 15:14:57.063312 (+ 858us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:57.063671 (+ 359us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:57.063701 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:57.064119 (+ 418us) log.cc:842] Serialized 26 byte log entry
0506 15:14:57.167893 (+103774us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:14:57.186232 (+ 18339us) log.cc:842] Serialized 14 byte log entry
0506 15:14:57.186746 (+ 514us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:57.186816 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:58.342526 (+1155710us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:14:58.363613 (+ 21087us) raft_consensus.cc:1671] finished
0506 15:14:58.363712 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:58.363832 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18027,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":8197,"raft.queue_time_us":1615,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":12698,"spinlock_wait_cycles":90266368,"thread_start_us":103378,"threads_started":1,"wal-append.queue_time_us":194208}
I20240506 15:14:58.353284 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:58.364027 21027 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.374771 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.375499 20882 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:58.377979 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.379055 20882 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:58.382160 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:58.307318 22880 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.316947 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:58.365687 21023 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.220800 22584 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.427376 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:58.428046 20881 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.448036 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:58.454720 22941 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:58.408905 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.404103 21026 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:14:58.413076 20880 raft_consensus.cc:3037] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.474869 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.471951 20996 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.412700 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:14:58.382911 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 74536b2ff8754c03a925c7b726433266, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.482199 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 69590.9us
W20240506 15:14:58.458797 23010 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.552s user 0.005s sys 0.023s
I20240506 15:14:58.483450 20921 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.406936 21027 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:14:58.493074 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.492307 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.494025 20951 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:58.496412 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:58.496709 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.497582 20957 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:58.500702 20956 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 1.
I20240506 15:14:58.500888 20996 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.514977 20880 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:58.516220 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.516538 20996 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.518277 20877 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:58.518999 20996 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.520799 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.521692 20996 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.500208 22947 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:14:58.522707 22980 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:58.522918 22988 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:58.523378 22980 raft_consensus.cc:686] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:58.523442 22988 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:58.523933 22988 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:58.524323 22980 consensus_queue.cc:237] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.509907 22972 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:58.447317 22584 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.506238 22874 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:58.507586 22983 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.549496 20883 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:58.530169 22981 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:58.532377 22972 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:58.542495 22871 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 15:14:58.537418 22947 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:14:58.572562 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet dd8ce6e0da3e465f875171ac339f3b50, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.573196 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet dd8ce6e0da3e465f875171ac339f3b50, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.582439 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.587211 20883 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 2.
I20240506 15:14:58.559202 22972 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:58.569499 22871 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:14:58.597350 20953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.010s sys 0.026s
I20240506 15:14:58.556221 23022 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:14:58.599865 20953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.010s sys 0.027s
W20240506 15:14:58.598454 20924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.023s sys 0.014s
W20240506 15:14:58.601775 20924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.023s sys 0.014s
I20240506 15:14:58.570044 22981 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:14:58.602396 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 546073us
I20240506 15:14:58.603246 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:58.604190 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.604889 20883 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:58.600418 22871 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:58.610957 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:58.601529 22557 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.616474 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:58.616045 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:58.618173 20883 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:58.619225 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:58.619822 20874 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.616487 22943 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:14:58.626621 22979 consensus_queue.cc:1046] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:14:58.627123 22943 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.627882 22979 raft_consensus.cc:3032] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:58.627702 22943 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.629910 22943 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.633739 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.634217 20955 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:58.621912 23024 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.626804 22915 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:14:58.641173 23029 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.640157 22876 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:58.645323 20877 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:58.646034 20879 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:58.642861 22936 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:58.647353 22936 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:58.647795 22936 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:58.653653 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 115) took 1193 ms. Trace:
I20240506 15:14:58.655284 20881 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.654175 20953 rpcz_store.cc:276] 0506 15:14:57.459987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:57.987613 (+527626us) service_pool.cc:225] Handling call
0506 15:14:57.987920 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:57.988395 (+ 475us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:57.988420 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:14:58.016318 (+ 27898us) log.cc:842] Serialized 26 byte log entry
0506 15:14:58.600869 (+584551us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:14:58.600905 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:58.600972 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:14:58.653418 (+ 52446us) raft_consensus.cc:1671] finished
0506 15:14:58.653438 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:14:58.653594 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9387,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":112,"thread_start_us":572387,"threads_started":1,"wal-append.queue_time_us":617221}
I20240506 15:14:58.615804 23028 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 15:14:58.666428 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f5a9099e10624a769e45882e9b864930, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.635675 22979 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
W20240506 15:14:58.666920 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f5a9099e10624a769e45882e9b864930, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.667925 20874 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.667708 20955 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:58.670641 21654 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.618495 22557 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.621380 22947 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:58.704484 23023 log.cc:924] Time spent T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20240506 15:14:58.641710 23029 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.677466 22584 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.714672 22557 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.663213 22981 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.691421 22991 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:58.653466 23024 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.730670 22988 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.720563 22979 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:58.737723 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 995) took 1279 ms. Trace:
I20240506 15:14:58.741497 20954 rpcz_store.cc:276] 0506 15:14:57.457991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:57.941274 (+483283us) service_pool.cc:225] Handling call
0506 15:14:57.958175 (+ 16901us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:14:57.988287 (+ 30112us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:14:58.737454 (+749167us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b540038d950. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eec59 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:14:58.737651 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37868800}
I20240506 15:14:58.749135 22979 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:58.750350 23025 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240506 15:14:58.751065 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.757369 20949 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:58.757519 23024 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.760530 20953 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240506 15:14:58.760705 20956 raft_consensus.cc:1261] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:58.766130 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.767041 20951 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.768743 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:58.769613 20951 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.772958 20953 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:14:58.779793 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.780699 21027 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:14:58.740463 22991 raft_consensus.cc:686] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:14:58.788904 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.795642 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.796231 22943 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:58.796564 20954 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:14:58.798388 23036 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240506 15:14:58.796432 22891 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:14:58.799657 22891 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:58.801930 22988 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.827828 22871 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:58.838554 23008 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:14:58.839332 20996 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.820614 22943 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:58.840458 22943 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:58.841027 22997 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:14:58.838917 22871 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:58.841480 22997 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:58.841949 22997 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:58.848834 22991 consensus_queue.cc:237] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:58.856829 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 147228us
I20240506 15:14:58.857833 20996 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.858053 20845 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:58.862320 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.865051 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.870025 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:58.870999 21032 raft_consensus.cc:2354] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:58.875572 20954 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:58.877135 20996 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.882550 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.883262 20955 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 1 FOLLOWER]: Advancing to term 3
I20240506 15:14:58.867977 22779 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:58.884752 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:58.888020 21656 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:14:58.892328 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:58.897156 20958 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:58.898840 23035 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.217s user 0.002s sys 0.000s
I20240506 15:14:58.893741 20949 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:14:58.822376 22976 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20240506 15:14:58.914784 21023 raft_consensus.cc:2454] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:14:58.870083 22871 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20240506 15:14:58.903252 22779 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:58.917261 22688 raft_consensus.cc:2741] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240506 15:14:58.937916 20998 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:58.943523 22892 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:58.904479 23018 raft_consensus.cc:2781] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:58.905182 22947 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:58.904899 22521 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:58.927911 20955 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:58.919119 23043 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
W20240506 15:14:58.982280 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 1.25966e+06us
I20240506 15:14:58.985994 20996 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.965196 22779 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:58.993095 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:58.987074 20958 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:59.005270 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:58.984230 22557 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:59.000005 22857 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 15:14:59.001925 22584 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.011s
I20240506 15:14:58.993901 23018 raft_consensus.cc:686] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
W20240506 15:14:59.024195 23034 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.363s user 0.002s sys 0.000s
I20240506 15:14:58.990341 23043 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:58.993383 23028 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.000685 22821 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader election won for term 3
I20240506 15:14:58.976581 23008 consensus_queue.cc:1046] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240506 15:14:59.013352 22971 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.023552 22584 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:59.003857 23029 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:58.991091 22947 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.068660 22584 raft_consensus.cc:686] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.065241 23026 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.034045 23028 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.041345 22821 raft_consensus.cc:686] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.006744 22943 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.040565 23018 consensus_queue.cc:237] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:59.101660 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 96511.2us
I20240506 15:14:59.038102 22971 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.112124 22521 raft_consensus.cc:686] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.113867 22971 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.113178 22521 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:59.115792 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 114990us
I20240506 15:14:59.108428 22857 tablet_bootstrap.cc:492] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:14:59.173133 23008 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.184540 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:59.033387 23043 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.190876 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.191772 20880 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:59.188618 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.188829 20949 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:59.193091 20874 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:59.194384 20996 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:59.195355 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.196076 20877 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:59.095456 23026 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.200877 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.201763 20879 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
W20240506 15:14:59.203527 23047 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
I20240506 15:14:59.204516 20876 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:59.205523 23028 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:59.208374 23049 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.086s user 0.003s sys 0.000s
I20240506 15:14:59.211997 22943 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.213102 22947 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.215509 20880 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:59.220078 21032 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:14:59.191535 22887 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
W20240506 15:14:59.225853 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 232793us
I20240506 15:14:59.220099 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.222658 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.227195 20881 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 2 FOLLOWER]: Advancing to term 3
I20240506 15:14:59.227401 21023 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:14:59.227504 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.222790 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.222051 20874 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:59.178122 22881 consensus_queue.cc:1035] T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
W20240506 15:14:59.214390 22972 consensus_meta.cc:287] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.612s user 0.006s sys 0.006s
I20240506 15:14:59.192968 22584 consensus_queue.cc:237] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.243662 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.223229 23043 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.246677 23048 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:59.247243 23048 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:59.249339 23048 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:59.250139 23048 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.249260 22972 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.252557 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:14:59.234616 22857 ts_tablet_manager.cc:1367] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 1.395s user 0.036s sys 0.026s
I20240506 15:14:59.265656 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.265730 20881 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:14:59.267040 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:59.273159 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 78851us
I20240506 15:14:59.193101 23008 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.244319 22971 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:59.253825 23045 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.214118 23056 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20240506 15:14:59.273274 22939 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.300536 22939 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.302706 22939 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.303593 22888 raft_consensus.cc:3032] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:14:59.278589 23055 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.306349 23055 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:14:59.307190 23061 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.193s user 0.000s sys 0.003s
W20240506 15:14:59.308745 22936 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.660s user 0.007s sys 0.003s
I20240506 15:14:59.309171 22936 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.296185 20949 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:59.269882 22887 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:14:59.304009 22888 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.218017 22821 consensus_queue.cc:237] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.278154 22988 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.324818 22972 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.324923 23008 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.326112 23026 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.311162 22936 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.280501 22971 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.332875 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.276477 22947 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
W20240506 15:14:59.348685 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 103754us
I20240506 15:14:59.317263 22892 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.352012 21027 raft_consensus.cc:2396] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240506 15:14:59.354728 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:59.360242 20956 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:14:59.361549 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:59.362469 20957 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:59.365211 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:14:59.365855 20951 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.326218 22857 raft_consensus.cc:348] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.327900 23065 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:59.333703 22880 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:59.373472 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:14:59.320461 22997 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:14:59.403738 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 45648.6us
I20240506 15:14:59.406555 21025 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:14:59.349851 23056 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.364823 23026 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.374853 20956 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:14:59.351425 22971 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.410863 20949 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
W20240506 15:14:59.416785 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 217589us
I20240506 15:14:59.342053 22988 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:59.294148 23045 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.419905 21023 raft_consensus.cc:2354] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:14:59.413982 20996 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:59.368611 22892 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.415940 23024 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:14:59.430420 23063 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20240506 15:14:59.417515 22876 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20240506 15:14:59.451330 20845 leader_election.cc:398] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
W20240506 15:14:59.470384 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 56509us
I20240506 15:14:59.502943 20845 leader_election.cc:401] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20240506 15:14:59.503449 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 237835us
I20240506 15:14:59.504215 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.430399 22880 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:14:59.513406 20996 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:14:59.514056 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 62965.7us
I20240506 15:14:59.514365 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:14:59.412698 23064 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.091s
I20240506 15:14:59.433158 23055 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.518661 22880 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.382407 22888 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.487474 22881 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:14:59.533711 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.546944 21026 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:14:59.432775 22857 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:14:59.419150 23026 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.539649 23071 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:59.556083 22997 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.557808 23045 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.558470 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:14:59.540694 23068 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20240506 15:14:59.541635 23008 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:14:59.560089 23008 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.541579 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:59.509267 22847 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:59.526468 23072 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:14:59.537943 22584 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:14:59.545711 22988 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:14:59.417304 22887 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.561359 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.542382 23066 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.564823 21653 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
W20240506 15:14:59.546489 22941 consensus_meta.cc:287] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 1.082s user 0.013s sys 0.001s
I20240506 15:14:59.488055 23024 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.565388 22881 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:14:59.568637 23008 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:14:59.594324 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.429379 23056 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.600143 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.599006 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.574083 23071 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:14:59.607039 20883 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:59.608431 20878 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:14:59.586386 22876 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:59.613937 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.615644 23021 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20240506 15:14:59.615937 23043 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:59.618623 23024 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.593515 22584 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.619410 23076 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:14:59.620676 22880 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.593665 23066 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.634495 22584 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.637702 23026 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:14:59.612748 22779 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.605s user 0.006s sys 0.005s
I20240506 15:14:59.602072 22857 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:14:59.596529 22980 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.627346 20846 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.629597 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.597148 22881 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:14:59.633000 23072 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.630331 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:14:59.631428 22876 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.645804 22978 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:59.666823 22978 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:14:59.667239 22978 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.667104 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.669493 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:14:59.669734 23053 log.cc:924] Time spent T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.554s user 0.001s sys 0.001s
I20240506 15:14:59.670161 20879 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:59.604665 23071 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:14:59.584389 22847 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.672845 23071 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:14:59.685442 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:14:59.655140 20881 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.649149 20951 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:59.656162 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:59.664526 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.703059 20881 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
W20240506 15:14:59.703334 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 47290.9us
I20240506 15:14:59.706239 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.665331 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.666204 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.657555 23066 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.720170 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.661147 23079 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:59.727016 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.725171 20879 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:14:59.735608 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.647179 22779 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.712307 23029 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:14:59.816108 20875 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240506 15:14:59.818733 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 81284.1us
I20240506 15:14:59.820998 23056 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.822196 22987 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20240506 15:14:59.824570 22996 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:59.825218 22996 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:59.825685 22996 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.742556 23073 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:59.656980 22980 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.726745 23076 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:14:59.834025 20883 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:14:59.837174 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 99673.9us
I20240506 15:14:59.839423 20998 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:59.733688 22892 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:14:59.829249 23080 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20240506 15:14:59.742115 22957 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:59.811245 23029 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:14:59.836547 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:59.872594 20951 raft_consensus.cc:1261] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:14:59.873103 20846 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:14:59.874219 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.875824 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.876539 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.877123 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:14:59.880142 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:14:59.843230 23021 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240506 15:14:59.878752 22957 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:14:59.893779 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:14:59.872188 22921 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:14:59.909782 20879 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:14:59.878026 23065 raft_consensus.cc:2741] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20240506 15:14:59.887284 22957 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:14:59.916522 22891 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:14:59.909289 22943 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:14:59.906450 23042 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240506 15:14:59.907357 23084 log.cc:924] Time spent T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20240506 15:14:59.871403 22978 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:14:59.907066 23043 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:14:59.927731 23087 consensus_queue.cc:1035] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240506 15:14:59.904790 23070 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:14:59.928378 23045 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:59.916586 23057 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:14:59.920338 22957 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.917383 20779 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 7 to 8, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 8 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:59.928015 22876 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:14:59.883175 23055 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20240506 15:14:59.935089 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:14:59.922248 22921 raft_consensus.cc:686] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.950004 22857 ts_tablet_manager.cc:1398] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.674s user 0.002s sys 0.032s
W20240506 15:14:59.950425 22857 ts_tablet_manager.cc:1430] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Tablet startup took 690ms
I20240506 15:14:59.952526 21686 raft_consensus.cc:2379] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
W20240506 15:14:59.950707 22857 ts_tablet_manager.cc:1432] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Trace:
0506 15:14:54.863362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:54.863521 (+ 159us) service_pool.cc:225] Handling call
0506 15:14:55.042635 (+179114us) ts_tablet_manager.cc:903] T 74536b2ff8754c03a925c7b726433266 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
0506 15:14:55.870661 (+828026us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:14:56.693529 (+822868us) tablet_metadata.cc:641] Metadata flushed
0506 15:14:56.724534 (+ 31005us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:14:56.838725 (+114191us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:14:56.839031 (+ 306us) tablet_replica.cc:205] Creating consensus instance
0506 15:14:56.839361 (+ 330us) ts_tablet_manager.cc:958] Replica registered.
0506 15:14:56.839428 (+ 67us) inbound_call.cc:173] Queueing success response
0506 15:14:57.831463 (+992035us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0506 15:14:59.275713 (+1444250us) ts_tablet_manager.cc:1399] Starting tablet replica
0506 15:14:59.297970 (+ 22257us) tablet_replica.cc:259] Starting instrumentation
0506 15:14:59.298896 (+ 926us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":66012,"mutex_wait_us":2423,"raft.queue_time_us":10498,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":27200,"tablet-copy.queue_time_us":126148,"tablet-download-pool.queue_time_us":144,"tablet-download-pool.run_cpu_time_us":1051,"tablet-download-pool.run_wall_time_us":727302,"thread_start_us":273353,"threads_started":3}
I20240506 15:14:59.939666 22939 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240506 15:14:59.955765 20968 tablet_copy_service.cc:342] P ecededcd854842d4867fda9fa63a4b66: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-74536b2ff8754c03a925c7b726433266 received from {username='slave'} at 127.0.0.1:44378
I20240506 15:14:59.956207 20968 tablet_copy_service.cc:433] P ecededcd854842d4867fda9fa63a4b66: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-74536b2ff8754c03a925c7b726433266 on tablet 74536b2ff8754c03a925c7b726433266 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:14:59.956133 22921 consensus_queue.cc:237] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.958145 23076 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:14:59.910900 23029 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.960608 23029 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:14:59.908720 22971 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.964781 23057 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:14:59.965344 22880 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20240506 15:14:59.967873 23057 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:14:59.926921 23065 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240506 15:14:59.945312 23042 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240506 15:14:59.971750 20779 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 5 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 6 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:59.973623 20779 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 4 to 5, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:14:59.976740 22980 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:14:59.961602 23072 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:14:59.978332 20878 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:14:59.980857 23043 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:14:59.982007 20880 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:14:59.992507 20779 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 6 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 7 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:14:59.980064 23079 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:00.003541 20876 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:14:59.925897 22943 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.011159 21024 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:14:59.982986 22779 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.017540 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.015655 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.023617 20958 raft_consensus.cc:2354] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:14:59.963016 23045 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.068s
I20240506 15:15:00.034407 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.040741 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.041178 20883 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:00.041711 20949 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:00.008678 23042 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240506 15:15:00.009071 23021 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
W20240506 15:15:00.052736 23060 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.938s user 0.007s sys 0.006s
I20240506 15:15:00.048825 23079 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:00.050791 22880 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:00.054056 23071 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.021489 23043 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240506 15:15:00.060786 20996 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:15:00.076979 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e81c09e0400045b180379648eed9b219, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:00.077347 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e81c09e0400045b180379648eed9b219, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.085458 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.086695 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.087393 20883 raft_consensus.cc:2354] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:00.091447 21026 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240506 15:15:00.091958 22847 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.092245 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.092916 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.093712 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:00.094226 23088 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20240506 15:15:00.027562 22876 consensus_queue.cc:1046] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240506 15:15:00.095839 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.096975 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.098047 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:00.098824 23089 log.cc:924] Time spent T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20240506 15:15:00.099097 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.056607 22891 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:00.101406 22891 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.103395 22891 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:00.104161 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.088097 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.100111 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.123785 20876 raft_consensus.cc:2379] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:00.062909 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:00.128460 20874 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:00.143812 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.154595 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.158025 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:00.105230 22980 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:00.165951 21027 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:00.163551 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.075987 22957 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.178978 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:00.179572 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.181146 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.182761 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:00.185024 23093 log.cc:924] Time spent T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20240506 15:15:00.194309 23048 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.944s user 0.008s sys 0.003s
I20240506 15:15:00.194900 23048 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:00.195791 20879 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:00.199533 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.199743 22971 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.200402 20880 raft_consensus.cc:2379] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:00.201206 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.201889 20880 raft_consensus.cc:2354] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:00.201838 23071 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:00.202817 22881 consensus_meta.cc:287] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.002s
I20240506 15:15:00.202911 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.203359 22881 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.204142 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.205281 20880 raft_consensus.cc:2396] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:15:00.206686 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.207767 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:15:00.208006 23008 consensus_meta.cc:287] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.596s user 0.007s sys 0.000s
I20240506 15:15:00.067836 23079 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.208847 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.209470 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.166433 23073 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:00.210850 22943 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:00.213140 23091 log.cc:924] Time spent T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.264s user 0.002s sys 0.000s
I20240506 15:15:00.067250 22880 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.217913 21027 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:00.222777 20998 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:00.225440 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:00.184096 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.258091 20883 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:15:00.259878 20883 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:15:00.270952 23079 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:00.170188 22978 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.261440 20876 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:00.274513 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.262253 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.263325 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:00.267452 22881 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.231534 20879 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:00.258091 20880 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:00.268810 22957 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.257186 22943 raft_consensus.cc:3032] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:00.287807 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:00.287798 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:00.273944 22930 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:00.213166 23070 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:00.301272 21653 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.236936 23008 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.329594 20882 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:00.329700 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.330709 20950 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:00.322033 23101 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:00.323240 22880 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.313314 23048 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240506 15:15:00.307468 23082 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20240506 15:15:00.352954 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.353978 20951 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:00.327610 23097 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:00.355984 21025 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:00.356281 21030 raft_consensus.cc:1261] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240506 15:15:00.333449 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.328500 23055 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:00.284787 22996 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.296838 22943 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:00.336607 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:00.339991 20955 raft_consensus.cc:2396] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240506 15:15:00.346693 20845 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:00.312573 22930 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:00.357259 20957 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:00.366442 22779 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240506 15:15:00.368755 20952 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:00.369454 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.378571 20949 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:00.379272 20878 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:00.385622 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.389247 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.389904 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.390758 20880 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:00.386618 20957 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:00.375398 22930 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:00.375537 23038 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:00.405702 20950 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:00.412614 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.418078 20957 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:00.387264 23102 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.427940 20880 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:00.349613 23101 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.432646 20998 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:00.376884 21686 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:00.376607 23055 consensus_queue.cc:1035] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20240506 15:15:00.445096 20874 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:00.449510 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:00.446357 22921 raft_consensus.cc:2781] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:00.451619 20950 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:00.364840 23097 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.436707 23102 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.465957 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:00.473989 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.475140 20950 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:00.424435 22943 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.489302 20950 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:00.460228 22980 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:00.458359 22921 raft_consensus.cc:686] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:00.455860 22880 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:00.501106 20950 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:00.505326 21028 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:00.493566 23008 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.472565 22976 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240506 15:15:00.508601 22996 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.502987 22880 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:00.509934 22880 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:00.509865 22921 consensus_queue.cc:237] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.511721 22991 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:00.512213 22991 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:00.516063 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:00.522958 23107 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240506 15:15:00.523358 22943 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:15:00.524816 23108 log.cc:924] Time spent T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240506 15:15:00.526510 21686 raft_consensus.cc:1226] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
W20240506 15:15:00.527891 23106 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20240506 15:15:00.535880 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.536589 20953 raft_consensus.cc:3037] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:00.540987 20952 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240506 15:15:00.544530 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.548758 20953 raft_consensus.cc:2454] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:00.559458 20958 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 15:15:00.561447 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.569343 20951 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:00.574165 20921 leader_election.cc:398] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:00.574609 20921 leader_election.cc:401] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240506 15:15:00.577037 20951 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:00.585817 20877 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:00.575574 23103 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:00.607913 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:00.620153 20957 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:00.623212 23105 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:00.626225 23105 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.589318 23070 consensus_queue.cc:1046] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20240506 15:15:00.628271 22991 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.636126 23102 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.637362 23097 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.639808 22880 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.640914 23101 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.683025 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.685930 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.687585 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.653453 23008 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:00.689446 23105 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.693584 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:00.695461 20996 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:00.657270 22821 consensus_queue.cc:1046] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20240506 15:15:00.710774 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.712179 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:00.704902 22557 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:00.702021 23008 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:15:00.711838 23115 log.cc:924] Time spent T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20240506 15:15:00.716761 22557 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:00.725023 22557 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:00.729017 22880 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:00.731153 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:00.718292 22688 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:00.732645 22688 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:00.733109 22688 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:00.734592 22888 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:00.737392 20996 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:00.738802 23008 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.708182 22980 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20240506 15:15:00.738667 22976 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:00.718526 22971 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240506 15:15:00.742986 20924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.042s sys 0.028s
W20240506 15:15:00.743312 20924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.042s sys 0.028s
W20240506 15:15:00.743526 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 716733us
I20240506 15:15:00.744339 20846 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:00.750980 23097 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.751590 23097 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:00.753674 23097 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:00.755690 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:00.746016 22971 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:00.756568 22971 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:00.755326 22930 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.756886 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:00.741928 23105 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.789680 23105 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.794441 22557 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.778055 20951 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:00.772986 23102 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.747804 22821 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:00.760713 23021 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240506 15:15:00.780525 20950 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:00.807699 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.781561 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.773427 22936 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:00.784019 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.760461 22941 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20240506 15:15:00.795042 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.798028 22983 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.799719 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.791661 23105 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.811676 20951 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:00.824239 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.813918 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.825536 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.826613 20951 raft_consensus.cc:2362] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5. Re-sending same reply.
I20240506 15:15:00.773578 23097 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:15:00.757469 23104 log.cc:924] Time spent T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.315s user 0.002s sys 0.000s
I20240506 15:15:00.787933 22688 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.812448 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.821826 20924 leader_election.cc:398] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:00.833600 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.825881 20952 raft_consensus.cc:2396] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240506 15:15:00.835551 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.836472 20952 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:00.837709 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.839509 20952 raft_consensus.cc:2379] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:00.827816 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.841068 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.828027 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.818032 20950 raft_consensus.cc:2354] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:15:00.842643 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.843091 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.845211 20924 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.796146 22971 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.845944 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:00.772179 22988 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.057s
I20240506 15:15:00.848150 20951 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240506 15:15:00.829828 22943 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:00.824877 22821 raft_consensus.cc:686] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:00.851264 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.855906 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.813249 23102 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.830420 22936 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:00.853853 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:00.850098 23105 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:00.852893 23101 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:00.851924 23103 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:00.830844 22983 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.840683 22584 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20240506 15:15:00.838908 23097 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.889796 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:00.888480 23102 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.909308 21686 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:00.868510 23103 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.954051 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1072) took 1376 ms. Trace:
I20240506 15:15:00.954519 21029 rpcz_store.cc:276] 0506 15:14:59.577865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:59.580764 (+ 2899us) service_pool.cc:225] Handling call
0506 15:14:59.581199 (+ 435us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:14:59.587057 (+ 5858us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:14:59.587101 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:00.217750 (+630649us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b2c00177f18. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630ccd1f11 00007f630cce1542 00007f630e32afc1 00007f630e2ef7da 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782
0506 15:15:00.233131 (+ 15381us) log.cc:842] Serialized 26 byte log entry
0506 15:15:00.355718 (+122587us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:15:00.355753 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:00.355844 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:00.942538 (+586694us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b2400444188. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b220024 00007f630b21e13b 00007f630b2ac149 00007f630b2abdd5 00007f630b2ab330 00007f630df9e3a8 00007f630df9eb08 00007f630df76618 00007f630df71a1a 00007f630df70e66
0506 15:15:00.953819 (+ 11281us) raft_consensus.cc:1671] finished
0506 15:15:00.953842 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:00.953990 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5317,"raft.queue_time_us":5877,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":3491,"spinlock_wait_cycles":85400832,"thread_start_us":122308,"threads_started":1,"wal-append.queue_time_us":174339}
I20240506 15:15:00.958177 22930 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.958315 22688 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.962250 23103 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.928025 22821 consensus_queue.cc:237] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.942560 22557 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:00.985970 23097 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:00.928232 22936 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:00.993221 20955 raft_consensus.cc:1226] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
W20240506 15:15:00.996806 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 145599us
I20240506 15:15:00.997684 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:00.980182 22971 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:01.001647 23133 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20240506 15:15:01.002189 23099 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.579s user 0.005s sys 0.001s
I20240506 15:15:01.024427 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:01.025162 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:01.025805 21032 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:15:01.026468 23139 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:01.027925 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:01.032399 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:01.033020 20958 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.032442 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:01.036573 22991 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:01.028364 23139 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:01.061542 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 63947.6us
I20240506 15:15:01.062393 20924 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.063551 20924 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.064891 20958 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:01.066718 20958 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240506 15:15:01.071919 20921 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:01.067173 20958 pending_rounds.cc:76] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 3
I20240506 15:15:01.074119 20958 pending_rounds.cc:97] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240506 15:15:01.075508 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:01.076467 20957 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.077487 20958 raft_consensus.cc:2866] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:01.071570 22943 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.073179 23024 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.073813 22876 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.094206 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.124069 20957 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
W20240506 15:15:01.126415 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 54575.8us
I20240506 15:15:01.134660 20996 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:01.146059 22930 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
W20240506 15:15:01.152743 23131 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.265s user 0.004s sys 0.000s
I20240506 15:15:01.153666 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:01.125708 23103 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.140888 23142 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:01.171705 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:01.166167 22930 raft_consensus.cc:686] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:01.167327 23139 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:01.177527 23142 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:01.199690 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 592589us
W20240506 15:15:01.198153 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 104050us
I20240506 15:15:01.206029 20924 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.207408 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.182483 22936 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:01.210080 21032 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:01.210688 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.212419 22930 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.213196 22983 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:01.217101 23071 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240506 15:15:01.218067 23146 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240506 15:15:01.219487 23143 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20240506 15:15:01.204967 23142 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.220994 23070 raft_consensus.cc:3032] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:01.221395 23070 raft_consensus.cc:729] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:01.222235 23070 consensus_queue.cc:260] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:01.223284 20876 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240506 15:15:01.226164 23048 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240506 15:15:01.226297 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 482011us
I20240506 15:15:01.227557 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:01.228395 23048 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:01.228906 23048 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:01.229542 22936 raft_consensus.cc:686] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:01.229785 23048 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:01.230474 22936 consensus_queue.cc:237] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.231092 20846 leader_election.cc:398] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:15:01.231412 20846 leader_election.cc:401] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240506 15:15:01.231621 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.229588 23139 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.232565 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.233173 23045 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:15:01.233520 22980 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:01.220435 22996 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:01.234544 22980 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:01.234616 22996 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:01.235112 22980 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:01.237067 23045 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:01.237557 23045 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.237946 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.238517 23139 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:01.239125 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:01.239445 23045 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:01.240283 20924 leader_election.cc:398] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:01.232416 20881 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:01.241091 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.241449 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:01.241657 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.241928 23024 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.230849 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.243525 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:01.244213 20876 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.244357 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.244985 20880 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:15:01.245918 23155 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:01.247133 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.247902 23155 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:15:01.249112 20924 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.249930 22880 consensus_queue.cc:1046] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:01.250487 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:01.235960 22980 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.251616 22891 raft_consensus.cc:3032] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:01.251720 22980 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.252097 22891 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:01.254604 22891 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.255959 22891 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.259750 23005 raft_consensus.cc:3032] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:01.272446 23005 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:01.273711 23005 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.275074 23005 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.235633 22996 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.242023 20881 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.240164 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:01.219974 23071 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:01.297209 23071 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:01.299841 23038 consensus_queue.cc:1046] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:01.301064 23098 raft_consensus.cc:3032] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:01.285295 22996 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.301479 23098 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:01.302424 23098 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.299870 23071 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:01.304419 23098 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.307713 23070 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.311218 21027 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:01.314556 23155 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240506 15:15:01.315750 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 66714us
I20240506 15:15:01.315765 22988 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:01.316680 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.317445 23152 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.318775 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.320502 23103 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.321816 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.336958 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:00.905263 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:01.349177 21027 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:01.353819 20998 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.355451 20881 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
W20240506 15:15:01.368629 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 46884.5us
I20240506 15:15:01.394702 20883 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:01.413002 20924 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:01.414666 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60950.9us
I20240506 15:15:01.421094 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:01.421877 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.429376 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:01.430181 21032 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:01.415439 20876 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:01.434021 20876 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:01.436874 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.441488 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.442541 20883 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:01.280638 23142 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.452051 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.458328 20924 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:01.377173 23103 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.421100 23026 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.470893 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:01.466372 20846 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.474555 20846 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.474524 23048 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.475904 23048 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:01.429427 23139 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:01.475972 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.488838 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:01.467639 23108 log.cc:924] Time spent T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240506 15:15:01.439956 22821 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.498178 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:01.457212 23145 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:01.499719 20949 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:01.502059 20955 raft_consensus.cc:1261] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:01.469026 23135 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:01.430292 23024 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20240506 15:15:01.469579 23097 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:01.470137 23152 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.480192 23101 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:01.518711 20955 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:01.494706 22871 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.458408 23071 consensus_queue.cc:1046] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20240506 15:15:01.430640 23155 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.093s
I20240506 15:15:01.513908 23135 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:01.575616 23142 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:01.581692 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 80225.5us
W20240506 15:15:01.568153 23176 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20240506 15:15:01.587412 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 157591us
I20240506 15:15:01.573297 23135 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:01.615440 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:01.611130 23157 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.156s user 0.002s sys 0.000s
W20240506 15:15:01.631242 23166 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20240506 15:15:01.658752 23169 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.235s user 0.003s sys 0.000s
W20240506 15:15:01.663215 23175 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.242s user 0.001s sys 0.000s
W20240506 15:15:01.682042 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 66289.6us
W20240506 15:15:01.686733 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 217735us
W20240506 15:15:01.690335 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 214409us
I20240506 15:15:01.691735 20846 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:01.699728 23145 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:01.688431 23155 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:01.713874 23005 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:01.722733 20956 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:01.727700 20952 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:01.728312 20955 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240506 15:15:01.790158 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 60920.8us
W20240506 15:15:01.792089 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 64945.8us
W20240506 15:15:01.803300 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 75296.8us
I20240506 15:15:01.703976 22881 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20240506 15:15:01.819761 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1347) took 1028 ms. Trace:
I20240506 15:15:01.820341 20953 rpcz_store.cc:276] 0506 15:15:00.792305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.072026 (+279721us) service_pool.cc:225] Handling call
0506 15:15:01.072361 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:01.072563 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:01.072586 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:01.075520 (+ 2934us) log.cc:842] Serialized 26 byte log entry
0506 15:15:01.435241 (+359721us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:15:01.435274 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:01.435335 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:01.819459 (+384124us) raft_consensus.cc:1671] finished
0506 15:15:01.819490 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:01.819670 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":352908,"threads_started":1,"wal-append.queue_time_us":372941}
I20240506 15:15:01.829309 22983 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20240506 15:15:01.797933 23155 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.853773 20952 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:01.815703 23019 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20240506 15:15:01.818817 23090 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20240506 15:15:01.868471 23021 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20240506 15:15:01.887171 21027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:01.888126 21027 raft_consensus.cc:595] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:01.904990 23155 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:01.905530 21029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:01.906183 21029 raft_consensus.cc:595] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:01.908653 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:01.911075 21024 consensus_queue.cc:237] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:01.917953 23135 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.919639 21032 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:01.938194 21027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:01.938882 21027 raft_consensus.cc:595] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:15:01.943601 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 354923us
I20240506 15:15:01.946252 22881 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:01.946534 23135 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:01.927995 23048 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:01.955009 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:01.956085 20954 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:01.946869 22881 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.953136 23038 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:01.953523 23048 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:01.982345 23048 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:01.992635 21024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:01.993301 21024 raft_consensus.cc:595] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:15:01.998422 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 306702us
I20240506 15:15:01.999981 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.008970 20846 leader_election.cc:398] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5.
I20240506 15:15:02.010443 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:02.010794 20996 consensus_peers.cc:485] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b540a7832fba49b5b2623616240c2711. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:15:02.012401 21024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:02.013208 21024 raft_consensus.cc:595] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:02.013672 21024 raft_consensus.cc:603] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:02.024523 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:02.020373 23005 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.041404 20846 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.048933 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1348) took 1256 ms. Trace:
I20240506 15:15:02.051049 20958 rpcz_store.cc:276] 0506 15:15:00.792685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.078710 (+286025us) service_pool.cc:225] Handling call
0506 15:15:01.079227 (+ 517us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:15:01.152414 (+ 73187us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:01.152462 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:15:01.153029 (+ 567us) log.cc:842] Serialized 52 byte log entry
0506 15:15:01.508425 (+355396us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:02.020695 (+512270us) log.cc:842] Serialized 14 byte log entry
0506 15:15:02.048331 (+ 27636us) log.cc:842] Serialized 14 byte log entry
0506 15:15:02.048552 (+ 221us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:02.048633 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:02.048692 (+ 59us) raft_consensus.cc:1671] finished
0506 15:15:02.048708 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.048857 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":576193,"prepare.queue_time_us":536195,"raft.queue_time_us":71157,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":5830,"thread_start_us":354988,"threads_started":1,"wal-append.queue_time_us":514036}
I20240506 15:15:02.067997 23048 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:15:02.127171 23083 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.128225 23083 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.133822 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:02.135383 23083 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.056771 23038 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:02.042999 23098 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240506 15:15:02.042017 23005 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.086448 22891 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240506 15:15:02.145443 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 104122us
I20240506 15:15:02.152233 20846 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.114467 22936 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:02.160589 23173 log.cc:924] Time spent T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.550s user 0.008s sys 0.005s
I20240506 15:15:02.161686 20846 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.162874 20846 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.163381 20951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1355) took 1141 ms. Trace:
I20240506 15:15:02.163926 23070 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.165858 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.167171 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.167383 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 70) took 5131 ms. Trace:
I20240506 15:15:02.174983 23038 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.150936 23071 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.181994 23038 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.184711 23038 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.168529 23207 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:02.170318 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.167999 23043 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:02.188880 22891 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240506 15:15:02.191525 23183 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.329s user 0.001s sys 0.001s
I20240506 15:15:02.193171 22881 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:02.194474 22936 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:02.180892 23164 log.cc:924] Time spent T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.764s user 0.005s sys 0.009s
I20240506 15:15:02.196782 22936 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:02.197378 20950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1349) took 1380 ms. Trace:
I20240506 15:15:02.197459 23204 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:02.198911 23005 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:02.197928 20950 rpcz_store.cc:276] 0506 15:15:00.816723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.091569 (+274846us) service_pool.cc:225] Handling call
0506 15:15:01.092027 (+ 458us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:01.126049 (+ 34022us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:01.126098 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:01.141690 (+ 15592us) log.cc:842] Serialized 26 byte log entry
0506 15:15:01.417491 (+275801us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:01.463394 (+ 45903us) log.cc:842] Serialized 14 byte log entry
0506 15:15:01.500566 (+ 37172us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:01.500634 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:02.179277 (+678643us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b1800335740. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b392b06 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441
0506 15:15:02.180221 (+ 944us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:02.197184 (+ 16963us) raft_consensus.cc:1671] finished
0506 15:15:02.197206 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.197319 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":505962,"prepare.queue_time_us":51540,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":37298,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":6712,"spinlock_wait_cycles":61902464,"thread_start_us":237995,"threads_started":1,"wal-append.queue_time_us":251685}
I20240506 15:15:02.201952 22996 raft_consensus.cc:3032] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:02.202447 22996 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:02.203236 22996 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.205986 20950 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:15:02.206197 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54786 (request call id 192) took 6648 ms. Trace:
I20240506 15:15:02.176908 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:02.207998 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:02.204401 21027 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:15:02.163841 20951 rpcz_store.cc:276] 0506 15:15:01.021669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.197438 (+175769us) service_pool.cc:225] Handling call
0506 15:15:01.209729 (+ 12291us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:01.210403 (+ 674us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:01.210431 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:01.210839 (+ 408us) log.cc:842] Serialized 26 byte log entry
0506 15:15:01.486207 (+275368us) raft_consensus.cc:1631] Marking committed up to 1
0506 15:15:01.486270 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:01.486354 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:02.159943 (+673589us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:02.163061 (+ 3118us) raft_consensus.cc:1671] finished
0506 15:15:02.163083 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.163314 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":382,"raft.queue_time_us":1032,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":49845120,"thread_start_us":236739,"threads_started":1,"wal-append.queue_time_us":310864}
I20240506 15:15:02.218495 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.220940 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.221654 20950 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:02.236783 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:02.237083 23191 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.237558 23043 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.237913 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.238003 23191 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.238724 22996 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:02.239133 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.242352 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.242875 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.243055 20877 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 3.
I20240506 15:15:02.240243 23191 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:02.241681 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.241892 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.240994 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.247831 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.238760 20951 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:02.251801 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:02.248673 20882 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:02.206616 20780 rpcz_store.cc:276] 0506 15:14:55.558209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:55.560390 (+ 2181us) service_pool.cc:225] Handling call
0506 15:15:02.206132 (+6645742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:14:57.565272 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:14:57.565989 (+ 717us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:14:57.566033 (+ 44us) write_op.cc:435] Acquired schema lock
0506 15:14:57.566059 (+ 26us) tablet.cc:660] Decoding operations
0506 15:14:57.687353 (+121294us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:14:57.701711 (+ 14358us) write_op.cc:641] Partition lock acquired for write op
0506 15:14:57.703295 (+ 1584us) tablet.cc:683] Acquiring locks for 23 operations
0506 15:14:57.974344 (+271049us) tablet.cc:699] Row locks acquired
0506 15:14:57.974378 (+ 34us) write_op.cc:260] PREPARE: finished
0506 15:14:57.974583 (+ 205us) write_op.cc:270] Start()
0506 15:14:57.974745 (+ 162us) write_op.cc:276] Timestamp: P: 1715008497974523 usec, L: 0
0506 15:14:57.974776 (+ 31us) op_driver.cc:331] REPLICATION: starting
0506 15:14:58.012546 (+ 37770us) log.cc:842] Serialized 9077 byte log entry
0506 15:14:58.634982 (+622436us) op_driver.cc:444] REPLICATION: finished
0506 15:14:58.946302 (+311320us) write_op.cc:301] APPLY: starting
0506 15:14:58.946412 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0506 15:14:58.946992 (+ 580us) tablet.cc:1368] finished BulkCheckPresence
0506 15:14:58.947170 (+ 178us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:14:58.950296 (+ 3126us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:14:58.951083 (+ 787us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0506 15:14:58.951163 (+ 80us) write_op.cc:312] APPLY: finished
0506 15:14:58.951771 (+ 608us) log.cc:842] Serialized 157 byte log entry
0506 15:14:58.980977 (+ 29206us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:58.996119 (+ 15142us) write_op.cc:454] Released schema lock
0506 15:14:59.039995 (+ 43876us) write_op.cc:341] FINISH: Updating metrics
0506 15:14:59.087827 (+ 47832us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:14:59.091477 (+ 3650us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":628227,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":289184,"apply.run_cpu_time_us":8021,"apply.run_wall_time_us":165175,"num_ops":23,"prepare.queue_time_us":120588,"prepare.run_cpu_time_us":31324,"prepare.run_wall_time_us":518398,"raft.queue_time_us":315918,"raft.run_cpu_time_us":41276,"raft.run_wall_time_us":263183,"replication_time_us":644807,"thread_start_us":532951,"threads_started":4,"wal-append.queue_time_us":76089,"wal-append.run_cpu_time_us":17102,"wal-append.run_wall_time_us":1920138}]]}
I20240506 15:15:02.256486 20879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1462) took 1027 ms. Trace:
I20240506 15:15:02.258536 20879 rpcz_store.cc:276] 0506 15:15:01.229213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.241153 (+ 11940us) service_pool.cc:225] Handling call
0506 15:15:01.665397 (+424244us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b58002c4998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d72e 0000000000339c39 00007f63131bc6c0 00007f63132823c4 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:15:01.665627 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:01.665908 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 15:15:01.687666 (+ 21758us) log.cc:842] Serialized 14 byte log entry
0506 15:15:02.256132 (+568466us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:02.256180 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:15:02.256232 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:02.256316 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.256429 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":219639,"prepare.queue_time_us":23064,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":282423,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":88508800,"thread_start_us":342323,"threads_started":2,"wal-append.queue_time_us":179146}
I20240506 15:15:02.265587 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.267253 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:02.268010 20877 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:15:02.270255 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.269865 21653 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:02.269219 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:02.274061 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.257934 23048 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:02.279526 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 334426us
I20240506 15:15:02.167830 20779 rpcz_store.cc:276] 0506 15:14:57.035511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:57.035737 (+ 226us) service_pool.cc:225] Handling call
0506 15:15:02.167314 (+5131577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:00.631723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:00.632296 (+ 573us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:00.632343 (+ 47us) write_op.cc:435] Acquired schema lock
0506 15:15:00.632361 (+ 18us) tablet.cc:660] Decoding operations
0506 15:15:00.633404 (+ 1043us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:00.633478 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:00.633510 (+ 32us) tablet.cc:683] Acquiring locks for 4 operations
0506 15:15:00.633895 (+ 385us) tablet.cc:699] Row locks acquired
0506 15:15:00.633909 (+ 14us) write_op.cc:260] PREPARE: finished
0506 15:15:00.634055 (+ 146us) write_op.cc:270] Start()
0506 15:15:00.634163 (+ 108us) write_op.cc:276] Timestamp: P: 1715008500634007 usec, L: 0
0506 15:15:00.634181 (+ 18us) op_driver.cc:331] REPLICATION: starting
0506 15:15:00.634836 (+ 655us) log.cc:842] Serialized 1685 byte log entry
0506 15:15:00.775705 (+140869us) op_driver.cc:444] REPLICATION: finished
0506 15:15:00.887835 (+112130us) write_op.cc:301] APPLY: starting
0506 15:15:00.887947 (+ 112us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:00.888147 (+ 200us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:00.888321 (+ 174us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:00.942347 (+ 54026us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:00.950137 (+ 7790us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0506 15:15:00.950184 (+ 47us) write_op.cc:312] APPLY: finished
0506 15:15:00.952671 (+ 2487us) log.cc:842] Serialized 40 byte log entry
0506 15:15:00.952874 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:00.952971 (+ 97us) write_op.cc:454] Released schema lock
0506 15:15:00.953020 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:00.953295 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:00.953352 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1355000,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":108510,"apply.run_cpu_time_us":3215,"apply.run_wall_time_us":65864,"num_ops":4,"prepare.queue_time_us":218870,"prepare.run_cpu_time_us":34401,"prepare.run_wall_time_us":84757,"raft.queue_time_us":25238,"raft.run_cpu_time_us":28548,"raft.run_wall_time_us":79093,"replication_time_us":141436,"thread_start_us":276051,"threads_started":4,"wal-append.queue_time_us":107823,"wal-append.run_cpu_time_us":10006,"wal-append.run_wall_time_us":1218729}]]}
I20240506 15:15:02.258625 23101 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.237597 21686 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:02.285749 20845 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.286118 20924 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.289731 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.292053 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.296499 23102 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.291038 22943 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:02.298501 22943 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.257288 23090 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20240506 15:15:02.288549 23191 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:02.288385 22876 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:02.307974 21687 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240506 15:15:02.315470 20877 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:02.322147 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:02.329272 20878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1463) took 1090 ms. Trace:
I20240506 15:15:02.287377 22929 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:02.331692 20924 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.335494 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:02.329722 20878 rpcz_store.cc:276] 0506 15:15:01.229499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.248392 (+ 18893us) service_pool.cc:225] Handling call
0506 15:15:01.248709 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:01.248893 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 4
0506 15:15:01.428894 (+180001us) log.cc:842] Serialized 14 byte log entry
0506 15:15:02.319087 (+890193us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:02.319148 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:02.319201 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:02.319274 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.319391 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":302566,"prepare.queue_time_us":107216,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":356601,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":17205,"thread_start_us":785160,"threads_started":3,"wal-append.queue_time_us":402500}
I20240506 15:15:02.319062 23201 raft_consensus.cc:2741] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240506 15:15:02.347213 20879 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:02.348026 20881 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:02.351408 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.352411 20881 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:02.319736 22876 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:02.349650 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:02.354321 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.354339 20845 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.357204 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:02.359555 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.361164 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.355315 20882 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:15:02.327342 23206 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:02.373422 20883 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:02.374353 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.375258 20876 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:02.390655 20881 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:02.376235 20877 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:02.337258 23103 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:02.339259 23142 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:02.337631 23144 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:15:02.386880 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 62202.5us
I20240506 15:15:02.326833 23098 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20240506 15:15:02.379568 23102 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:02.361891 20879 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:15:02.411808 20882 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240506 15:15:02.426919 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:02.409013 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:02.378530 23019 consensus_queue.cc:1035] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240506 15:15:02.403200 22943 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:02.402081 23103 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:02.445747 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 91574.3us
I20240506 15:15:02.372480 22876 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.424827 23142 raft_consensus.cc:686] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:02.483304 21656 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.489574 23196 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:02.490051 23196 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:02.490379 23196 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:02.407321 22881 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20240506 15:15:02.459157 23103 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:02.437405 23102 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.510789 23021 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:02.509693 23189 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.522408 23144 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:02.540690 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 47590.4us
W20240506 15:15:02.546254 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 90366.7us
I20240506 15:15:02.531692 23026 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:15:02.554258 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 145439us
W20240506 15:15:02.557699 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 100588us
I20240506 15:15:02.539614 23021 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:02.568193 22821 raft_consensus.cc:979] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:02.574628 20845 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:02.596478 22891 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:02.600924 23045 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:02.560856 23142 consensus_queue.cc:237] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.573259 23021 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:02.545650 23189 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:02.610740 23218 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240506 15:15:02.615979 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:02.546523 23144 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.599239 23191 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:02.626341 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.633033 20882 raft_consensus.cc:2379] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:15:02.635093 23209 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20240506 15:15:02.762271 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:02.637168 22876 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:02.637424 23191 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:02.654541 23101 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.748394 22821 raft_consensus.cc:1067] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:02.747938 22929 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:02.763317 23212 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.764950 23189 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.782625 23187 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.668303 23144 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.781047 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:02.793740 23098 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:02.837775 23211 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.469s user 0.002s sys 0.000s
W20240506 15:15:02.838714 23223 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
W20240506 15:15:02.839422 23219 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.274s user 0.000s sys 0.000s
I20240506 15:15:02.792874 23203 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.796715 23038 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:02.821010 22929 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:02.918018 23048 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.918735 23048 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.919116 20952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1392) took 1011 ms. Trace:
I20240506 15:15:02.920469 23213 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.921168 23213 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.923666 23213 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:02.851639 23098 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.928977 23026 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:02.814263 23101 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.929807 23098 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.929649 23026 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.829458 23212 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.931735 23026 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.929827 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:02.933251 23212 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.934007 23208 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.934743 23208 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.902930 23102 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.919822 20952 rpcz_store.cc:276] 0506 15:15:01.907602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.990079 (+ 82477us) service_pool.cc:225] Handling call
0506 15:15:01.990415 (+ 336us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 15:15:02.002323 (+ 11908us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:02.002363 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 15:15:02.019642 (+ 17279us) log.cc:842] Serialized 52 byte log entry
0506 15:15:02.143701 (+124059us) raft_consensus.cc:1631] Marking committed up to 2
0506 15:15:02.157183 (+ 13482us) log.cc:842] Serialized 14 byte log entry
0506 15:15:02.157805 (+ 622us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:02.158047 (+ 242us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:02.665902 (+507855us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:02.918611 (+252709us) raft_consensus.cc:1671] finished
0506 15:15:02.918788 (+ 177us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:02.919029 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1131,"prepare.queue_time_us":11815,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":50238,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":13608,"spinlock_wait_cycles":122840448,"thread_start_us":105926,"threads_started":1,"wal-append.queue_time_us":315966}
I20240506 15:15:02.929848 23048 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.942981 23101 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:02.952473 23182 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.848927 23187 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.963646 23070 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.973323 23212 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:02.976589 23098 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.976434 23213 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:02.912159 23038 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:02.942731 23203 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.999789 23071 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:02.981158 23196 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:03.001088 23070 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.000746 23212 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:02.994918 23182 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.016731 23205 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:03.044641 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 385410us
I20240506 15:15:03.006457 23098 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:03.047223 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 482497us
I20240506 15:15:03.009752 23213 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.049921 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.049726 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.040546 23202 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.050433 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.053620 23202 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.066921 20875 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
W20240506 15:15:03.062711 23103 consensus_meta.cc:287] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.534s user 0.009s sys 0.002s
I20240506 15:15:03.026866 23071 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:03.068946 23222 log.cc:924] Time spent T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20240506 15:15:03.056104 23202 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.068621 23208 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.125846 23212 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.126816 23203 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.127986 23196 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:03.152211 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.154212 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:03.158301 23070 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.160216 23182 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.155010 20956 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 9
I20240506 15:15:03.130631 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.133133 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.201885 20950 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:03.203116 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.066380 21686 raft_consensus.cc:2354] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9.
W20240506 15:15:03.134575 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 61261.8us
I20240506 15:15:03.136519 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.209118 20949 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:03.137604 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:03.166543 23235 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20240506 15:15:03.176998 23021 consensus_meta.cc:287] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.570s user 0.011s sys 0.000s
W20240506 15:15:03.185257 23233 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.195s user 0.001s sys 0.001s
I20240506 15:15:03.189652 22939 raft_consensus.cc:979] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:03.051712 23205 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.224999 23021 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.210130 23101 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:03.227281 23101 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:03.229620 23232 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.193s user 0.002s sys 0.000s
I20240506 15:15:03.229871 23187 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.121361 23142 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.125267 22981 consensus_queue.cc:1035] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240506 15:15:03.130313 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.137955 23102 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.242753 20956 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 9.
I20240506 15:15:03.255244 21654 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:03.257756 20776 catalog_manager.cc:5484] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.134778 23229 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20240506 15:15:03.261448 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.261673 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.260800 20776 catalog_manager.cc:5484] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20240506 15:15:03.268189 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 59088.6us
I20240506 15:15:03.272142 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:03.139448 23103 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.269186 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.277454 20776 catalog_manager.cc:5484] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:03.282334 20776 catalog_manager.cc:5484] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.280230 23103 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:03.281606 23215 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.781s user 0.003s sys 0.011s
I20240506 15:15:03.295248 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.296571 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.297467 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.295472 20922 leader_election.cc:398] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9.
I20240506 15:15:03.298444 20922 leader_election.cc:401] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:03.296635 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.299743 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1390) took 1401 ms. Trace:
I20240506 15:15:03.300791 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.301307 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.302325 21026 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:03.302891 23103 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9.
I20240506 15:15:03.300329 20954 rpcz_store.cc:276] 0506 15:15:01.898316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.957563 (+ 59247us) service_pool.cc:225] Handling call
0506 15:15:01.957963 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:01.960691 (+ 2728us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:01.960735 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:01.965603 (+ 4868us) log.cc:842] Serialized 26 byte log entry
0506 15:15:02.326829 (+361226us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:02.326865 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:02.326928 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:03.175906 (+848978us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:03.299316 (+123410us) raft_consensus.cc:1671] finished
0506 15:15:03.299352 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:03.299667 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10725,"raft.queue_time_us":2317,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":13112,"spinlock_wait_cycles":51442560,"thread_start_us":360410,"threads_started":1,"wal-append.queue_time_us":439935}
I20240506 15:15:03.294364 20776 catalog_manager.cc:5484] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.314787 20776 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.316710 23205 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.317921 23101 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.319769 22929 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.320966 22983 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:03.323343 22939 raft_consensus.cc:1067] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:03.325708 22929 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:03.327315 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55331.3us
I20240506 15:15:03.328960 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.332491 23226 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:03.333005 23226 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:03.333397 23226 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:03.335085 23191 consensus_meta.cc:287] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.529s user 0.010s sys 0.000s
I20240506 15:15:03.331208 20776 catalog_manager.cc:5484] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:03.337829 23230 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:03.266770 23102 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.340593 23021 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:03.341869 23098 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.343420 20776 catalog_manager.cc:5484] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 5, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.337062 22930 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:03.259323 23142 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.346483 20952 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:03.290144 23243 ts_tablet_manager.cc:902] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.337220 23071 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:03.335628 23191 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:03.350826 20776 catalog_manager.cc:5484] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 5, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.364701 23239 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.360023 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.362414 23142 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.364586 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.372718 21029 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:03.365531 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.362742 22929 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.332762 22888 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:03.348023 23055 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:03.374843 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.337906 23213 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.376888 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.380781 22929 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.381570 23055 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:03.383114 23055 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:03.380930 21024 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:03.370487 20776 catalog_manager.cc:5484] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:03.389923 21024 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:03.392019 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:03.393008 21026 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:03.397876 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.395645 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.400621 21024 raft_consensus.cc:2379] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:03.384064 22929 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:03.363903 23038 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:03.340106 23102 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:03.437081 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 302840us
I20240506 15:15:03.422076 21026 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:03.423882 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.438434 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:03.438796 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 309311us
I20240506 15:15:03.438458 21024 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:03.441089 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.441511 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:03.442499 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.444056 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.445389 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.446367 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.453004 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.358932 23226 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:03.451972 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.450050 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.450589 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.461613 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.464561 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.465312 20875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:03.465879 20875 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:03.466260 20875 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:03.374927 23239 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.467931 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:03.468740 20880 raft_consensus.cc:2379] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:03.469413 20955 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240506 15:15:03.470378 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.472820 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.475919 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.479103 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:03.480078 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:03.456496 23205 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.482939 20874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:03.483675 20874 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:03.484032 20874 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:03.462405 23144 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:03.467553 23142 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:03.525928 23142 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:03.526540 23142 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:03.529557 23191 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.471527 23103 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.531848 23103 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.476827 23155 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:03.538084 23226 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:03.551936 23189 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.552845 23189 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.555500 23189 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.489748 23205 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.558990 23205 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.492077 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.492660 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:03.483310 23145 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:03.493260 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:03.499676 20845 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:03.502457 23102 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.502213 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:03.511799 22876 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.513548 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:03.524502 23239 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:03.590245 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 90684.9us
I20240506 15:15:03.597991 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:03.596503 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 83184.9us
I20240506 15:15:03.600412 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.600502 20875 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.601992 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:03.602612 20875 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.603206 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.604542 23103 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.605100 23103 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.605257 20845 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.605607 23103 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.607673 23103 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.608368 23189 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:03.609434 20845 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.609777 20880 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:03.611092 22876 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.612797 23045 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:03.561347 23245 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:03.616860 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.596627 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:03.618454 20876 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:03.597051 23038 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:03.608273 23204 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:03.603631 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.611979 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.624881 20880 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:03.626276 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.633316 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.583968 20874 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.637032 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.639739 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:03.640156 21026 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:03.640458 21024 raft_consensus.cc:3032] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:03.640774 21024 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:03.641480 21024 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.642133 23142 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.491544 23144 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:03.643712 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:03.644634 20880 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:03.644897 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:03.646709 20874 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.647512 20875 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:03.648674 21032 raft_consensus.cc:2379] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:03.648557 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.653000 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:03.653898 20874 raft_consensus.cc:3037] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:03.649487 20875 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:03.672600 20875 raft_consensus.cc:1226] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
W20240506 15:15:03.674779 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 57990.9us
I20240506 15:15:03.675673 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.676405 20880 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 9.
I20240506 15:15:03.679527 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.680377 20880 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:03.681483 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:15:03.672487 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 54027us
I20240506 15:15:03.609791 23203 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:03.608320 23205 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:03.733443 23248 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:03.737443 23142 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.729750 20954 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:03.737972 23142 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.738461 23142 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.685559 23145 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.624387 23204 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:03.741075 23145 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.741802 23204 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:03.732415 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 56819.6us
I20240506 15:15:03.742944 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.743814 23145 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.648170 23144 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:03.756772 23145 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.757334 23145 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.759290 23145 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.759781 23145 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.733418 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:03.760707 21032 raft_consensus.cc:3032] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:03.761013 21032 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:03.624944 23038 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.761608 23038 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:03.761677 21032 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.762775 21032 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:03.725201 23103 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.764595 23103 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.672227 23196 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.734519 22876 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.735000 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:03.616510 23102 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.744385 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.785557 21689 raft_consensus.cc:2379] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 7: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:15:03.785945 21024 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:03.786319 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.745151 23249 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:03.744961 23247 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:03.745357 23045 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240506 15:15:03.642916 23071 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.747639 23019 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:03.644166 23155 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.710291 20876 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
W20240506 15:15:03.795154 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 108742us
I20240506 15:15:03.797194 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.798358 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:03.673697 23245 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.784765 23203 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:03.809480 20876 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:03.811367 23216 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:03.811833 21024 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:03.794150 23043 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:03.791142 23249 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.814194 20846 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:03.801023 23048 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:03.824388 20845 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:03.826112 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:03.815047 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 335076us
I20240506 15:15:03.799358 23202 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:03.837436 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:03.831681 20874 raft_consensus.cc:2454] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:15:03.760200 23145 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.803436 23247 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:03.805205 22891 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:03.843981 20996 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:03.793740 23196 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:03.838578 20921 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:03.836598 23202 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:03.823938 23142 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.796576 23071 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.802291 23155 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.824206 23103 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.847504 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:03.847540 21656 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.871644 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:03.833535 23249 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:03.884568 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.838704 23182 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:03.816167 23203 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.872200 23248 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:15:03.801422 23019 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.844063 23043 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:03.847047 23187 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:03.884721 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.843456 23101 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20240506 15:15:03.805955 23245 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.910889 23243 tablet_copy_client.cc:316] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:15:03.908776 23247 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.912097 23245 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.912222 23247 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.913491 23196 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.913697 23245 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.914419 23247 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.914237 23245 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.883533 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.915984 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.871702 22876 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:03.882951 23240 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:03.903194 23103 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:03.923955 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.875663 23186 raft_consensus.cc:2781] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:03.917699 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.919226 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.887626 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:03.919706 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.898012 23208 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.922057 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.873986 23246 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:03.900867 22847 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.921980 23248 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:03.928853 23102 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.929569 23135 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:03.941572 23145 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.945044 23019 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:03.945958 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet b540a7832fba49b5b2623616240c2711 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:15:03.946142 22876 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:03.961568 23246 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:03.947680 23203 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.949225 23071 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:03.924506 23103 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:03.915663 23212 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:03.960526 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.961074 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:03.991884 21032 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:03.946642 23142 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.017338 20924 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.959293 23249 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.020871 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:03.963680 23186 raft_consensus.cc:686] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:03.961778 23208 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:03.971151 22880 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20240506 15:15:03.916817 23202 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.018275 23103 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.000378 23246 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:03.971956 22847 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.002650 23254 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.002784 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet b540a7832fba49b5b2623616240c2711 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-b540a7832fba49b5b2623616240c2711
I20240506 15:15:04.075166 20876 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:04.080317 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.073246 23260 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.059036 23142 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.099225 20874 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:04.104238 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.067235 22876 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.105060 20874 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:04.107967 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.110535 20845 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:04.078850 23261 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:04.115433 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.115747 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:04.116685 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.117561 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.118319 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.091771 23102 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:04.092947 23245 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.056989 23248 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.093539 23155 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.147106 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.153187 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.154553 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.155685 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:04.158417 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.159098 20877 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:04.161911 23249 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.120888 22880 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:15:04.162681 22847 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.138993 23186 consensus_queue.cc:237] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:04.140636 23240 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:04.166291 23240 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:04.166741 23240 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.116112 23261 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.168550 23261 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.173192 23142 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.107314 23260 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.162674 22880 raft_consensus.cc:686] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:04.174990 22880 consensus_queue.cc:237] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.175622 23260 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.130280 23145 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:04.176496 23145 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:04.176987 23145 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.176766 20874 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20240506 15:15:04.183858 23257 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20240506 15:15:04.194559 23255 log.cc:924] Time spent T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20240506 15:15:04.199654 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.147806 23245 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.208977 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.186331 23155 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.187846 23142 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.103215 23254 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:04.210428 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 189603us
I20240506 15:15:04.215080 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.215869 20955 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.218904 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:04.220578 23253 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
I20240506 15:15:04.223383 23038 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.226693 23204 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.207732 23144 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.214689 23196 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:04.235713 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.242673 23254 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:04.245891 23256 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.188s user 0.002s sys 0.000s
I20240506 15:15:04.246431 23196 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240506 15:15:04.249703 23038 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.144812 23102 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.213418 23245 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.216169 23155 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:04.259816 23258 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
W20240506 15:15:04.260977 23259 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20240506 15:15:04.263388 23155 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.264320 23208 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.264993 23204 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.232071 23144 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.254231 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.280530 23254 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:04.283020 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.283891 20952 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:04.289209 20951 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:04.290058 23245 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.291755 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:04.301903 20955 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:04.307417 20924 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:04.219007 23142 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.316435 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.320585 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.322355 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.327105 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.309478 20878 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:04.331290 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.333177 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.334884 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.336088 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:04.338129 21024 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:04.339064 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.340036 21026 raft_consensus.cc:2379] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:04.341154 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:04.344659 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.305873 23098 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:04.350431 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.352280 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:04.352970 20950 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:04.348824 23271 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:04.357749 23142 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.357962 23271 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:04.358361 23271 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.354631 23261 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.366052 20956 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:04.368335 23261 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.369402 23102 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.373032 23103 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.374269 20954 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:04.375133 23202 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.375288 23103 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.383572 20921 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.390735 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.385293 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.392621 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.276063 23246 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.397812 20954 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:04.389269 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.344745 23101 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:04.400871 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.402364 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.403003 20950 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:15:04.403946 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.406785 20950 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:04.423179 20954 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:04.424902 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:04.441694 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.446954 21032 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.425961 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.448040 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.448947 20949 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:04.377301 23098 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.454109 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.455703 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.456513 20956 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.457839 20954 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:04.480419 20949 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240506 15:15:04.429476 23245 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:04.432705 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:04.488794 20957 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:15:04.427309 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 43845.5us
I20240506 15:15:04.435257 21027 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:04.489897 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.399976 23101 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.494127 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.495640 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.395519 23145 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.515673 23145 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.396116 23182 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:04.492422 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.516422 23182 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.492813 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.518373 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.518632 23182 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.505627 23101 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.519507 20955 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:04.523399 23048 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:04.523833 23048 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:04.506222 23103 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:04.508404 23135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:49614 (request call id 6) took 2602 ms. Trace:
I20240506 15:15:04.511423 23098 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.477541 23208 raft_consensus.cc:465] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Not starting pre-election -- already a leader
I20240506 15:15:04.511236 23271 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.513114 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.539964 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.513805 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:04.541105 20876 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:04.529057 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:04.486775 23263 log.cc:924] Time spent T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20240506 15:15:04.533434 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:04.534405 20957 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:04.529913 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.531920 21032 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:04.438311 23019 consensus_queue.cc:1046] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.087s
W20240506 15:15:04.536206 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 100628us
I20240506 15:15:04.546105 20877 raft_consensus.cc:2379] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:04.541196 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:04.546314 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.547320 20881 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:04.547680 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:04.560539 20924 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:04.561087 20924 leader_election.cc:401] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240506 15:15:04.565909 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.567345 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:04.568163 20952 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:04.569170 23261 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:04.571045 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.572103 23261 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:04.573884 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.578678 20846 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:04.515870 23240 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.540637 23082 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20240506 15:15:04.581488 23240 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:04.585088 20877 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:04.589017 20921 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.587047 20956 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:04.589783 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:04.591123 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.596729 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.600464 23103 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:04.601933 23271 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.594391 23261 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:04.605887 23101 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.605978 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.592640 22876 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:04.608233 23271 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:04.575693 23048 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:04.609071 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.609738 23048 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.610662 23271 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:04.611829 20921 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.595137 22880 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:04.595259 23204 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:04.613590 23071 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:15:04.613981 23071 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.614094 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:04.614405 23071 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:04.614859 23145 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:04.615494 23145 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:04.617245 23208 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:04.617764 23208 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:04.618194 23208 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:04.628216 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:04.632006 22930 raft_consensus.cc:3032] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:04.632704 22930 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:04.633786 22930 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:04.640560 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 43503.4us
W20240506 15:15:04.666561 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 52541.1us
I20240506 15:15:04.666942 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:04.667505 20921 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:04.645031 23276 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20240506 15:15:04.647228 23273 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240506 15:15:04.636193 22930 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:04.648188 23246 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:04.608072 23101 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:04.668612 23271 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:15:04.700848 23277 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.194s user 0.005s sys 0.000s
W20240506 15:15:04.749723 23278 log.cc:924] Time spent T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
W20240506 15:15:04.751677 23286 log.cc:924] Time spent T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
W20240506 15:15:04.759742 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 229344us
W20240506 15:15:04.764035 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 256107us
W20240506 15:15:04.759254 22929 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.006s sys 0.011s
W20240506 15:15:04.799806 22929 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.006s sys 0.011s
W20240506 15:15:04.753391 23275 log.cc:924] Time spent T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20240506 15:15:04.535543 23135 rpcz_store.cc:276] 0506 15:15:01.905657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:01.906931 (+ 1274us) service_pool.cc:225] Handling call
0506 15:15:01.914482 (+ 7551us) log.cc:842] Serialized 506 byte log entry
0506 15:15:02.162325 (+247843us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b180022a500. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3c51b8 00007f630b3c8691 00007f630b3c8649 00007f630b3c85d9 00007f630b3c85a1 00007f630b3c789d 00007f6314639da4 00007f6314639bd9 00007f630b3a7936
0506 15:15:02.305250 (+142925us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b180022a500. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3c52e4 00007f630b3c8691 00007f630b3c8649 00007f630b3c85d9 00007f630b3c85a1 00007f630b3c789d 00007f6314639da4 00007f6314639bd9 00007f630b3a7936
0506 15:15:04.508295 (+2203045us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22418,"raft.queue_time_us":31381,"raft.run_cpu_time_us":2162,"raft.run_wall_time_us":60885,"spinlock_wait_cycles":68454400,"thread_start_us":35015,"threads_started":1,"wal-append.queue_time_us":28023,"wal-append.run_cpu_time_us":1910,"wal-append.run_wall_time_us":1030912}
W20240506 15:15:04.953812 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 418863us
W20240506 15:15:04.999603 23202 consensus_meta.cc:287] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.624s user 0.007s sys 0.005s
I20240506 15:15:05.134300 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:05.143096 20882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.009s sys 0.009s
W20240506 15:15:05.320863 20882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.010s sys 0.010s
W20240506 15:15:05.157414 20958 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.007s sys 0.015s
W20240506 15:15:05.321998 20958 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s user 0.008s sys 0.015s
W20240506 15:15:05.321909 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 188207us
I20240506 15:15:05.322371 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:05.164500 20950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.017s sys 0.007s
I20240506 15:15:05.323552 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1444) took 1010 ms. Trace:
W20240506 15:15:05.324013 20950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.018s sys 0.007s
I20240506 15:15:05.204835 20921 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.045825 23071 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.323856 22880 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:05.327086 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 122779us
I20240506 15:15:05.258849 23101 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:05.328899 21030 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:05.328701 23101 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:05.330276 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.260968 23098 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.320113 23299 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:05.157430 20953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.038s sys 0.026s
W20240506 15:15:05.337334 20953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s user 0.039s sys 0.026s
I20240506 15:15:05.343318 23299 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.350550 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.351647 23098 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.354411 23098 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.355089 23098 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.355795 23098 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.357422 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1442) took 1057 ms. Trace:
I20240506 15:15:05.357892 20953 rpcz_store.cc:276] 0506 15:15:04.300375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:04.300807 (+ 432us) service_pool.cc:225] Handling call
0506 15:15:04.301094 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:04.301288 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:15:04.538023 (+236735us) log.cc:842] Serialized 14 byte log entry
0506 15:15:05.356849 (+818826us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:05.356949 (+ 100us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:05.357043 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:05.357166 (+ 123us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:05.357351 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":280538,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":1246,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":1035928,"threads_started":2,"wal-append.queue_time_us":802056}
I20240506 15:15:05.332398 23103 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:15:05.362016 23208 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.743s user 0.012s sys 0.002s
I20240506 15:15:05.362474 23102 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:05.324066 20958 rpcz_store.cc:276] 0506 15:15:04.312982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:04.313141 (+ 159us) service_pool.cc:225] Handling call
0506 15:15:04.325266 (+ 12125us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:04.325465 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 3
0506 15:15:04.532960 (+207495us) log.cc:842] Serialized 14 byte log entry
0506 15:15:05.322872 (+789912us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:05.322992 (+ 120us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:05.323101 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:05.323259 (+ 158us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:05.323472 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":319818,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":178,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":201,"thread_start_us":977166,"threads_started":2}
I20240506 15:15:05.366344 23299 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.369017 23071 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.380875 21028 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:05.366904 23208 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.383039 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:05.401274 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 73191.8us
I20240506 15:15:05.403898 20921 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.385444 23098 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.417455 23103 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:05.412447 23071 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:05.413941 23135 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:05.420611 23293 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.073s user 0.002s sys 0.001s
I20240506 15:15:05.424780 20875 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:05.427330 21042 tablet_copy_source_session.cc:205] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:15:05.420926 23101 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:05.436779 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 69110.6us
I20240506 15:15:05.437461 23083 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:05.438253 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.437115 23208 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.432775 23135 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.452656 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:05.454178 20881 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:05.455364 23291 log.cc:924] Time spent T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.110s user 0.002s sys 0.000s
I20240506 15:15:05.441712 23071 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:05.474625 23098 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:05.488802 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 85063.9us
I20240506 15:15:05.494058 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.496294 23098 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.496690 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.497913 23289 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.498291 23135 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:05.500101 23292 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20240506 15:15:05.500921 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:05.501839 20877 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.504154 20874 raft_consensus.cc:1226] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:05.505187 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:05.517756 20950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 1083) took 1120 ms. Trace:
I20240506 15:15:05.518383 21042 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:49620 (request call id 237) took 1606 ms. Trace:
I20240506 15:15:05.518965 21042 rpcz_store.cc:276] 0506 15:15:03.912426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:03.945647 (+ 33221us) service_pool.cc:225] Handling call
0506 15:15:05.327792 (+1382145us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001e2b54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f94a7 00007f631320d386 00007f6313216bfe 00007f630acf462b 00007f6313216b7a 00007f6313211c8c 00007f631320cfeb 00007f6313201bc2 00007f631230ba74 00007f631230ba02
0506 15:15:05.518231 (+190439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2524247808}
I20240506 15:15:05.520458 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:05.518326 20950 rpcz_store.cc:276] 0506 15:15:04.397628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:04.409467 (+ 11839us) service_pool.cc:225] Handling call
0506 15:15:04.409792 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:04.427992 (+ 18200us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:04.428035 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:04.439097 (+ 11062us) log.cc:842] Serialized 26 byte log entry
0506 15:15:05.324911 (+885814us) raft_consensus.cc:1631] Marking committed up to 8
0506 15:15:05.404590 (+ 79679us) log.cc:842] Serialized 14 byte log entry
0506 15:15:05.408552 (+ 3962us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:05.410237 (+ 1685us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:05.421230 (+ 10993us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:05.517369 (+ 96139us) raft_consensus.cc:1671] finished
0506 15:15:05.517496 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:05.517682 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18850,"prepare.queue_time_us":100923,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":22925,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":10203,"spinlock_wait_cycles":39080832,"thread_start_us":877644,"threads_started":2,"wal-append.queue_time_us":884707}
I20240506 15:15:05.526264 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:05.526335 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:05.527298 20874 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:05.527459 20876 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:05.530680 23071 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:05.531229 23071 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:05.531605 23071 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.533416 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.544397 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b540a7832fba49b5b2623616240c2711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:15:05.542150 20877 raft_consensus.cc:2454] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:05.544771 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.549549 23208 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:05.551187 23208 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:05.542150 23098 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.559516 20881 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:05.560827 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.562574 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.563915 23208 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:05.563547 22930 consensus_meta.cc:287] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.860s user 0.003s sys 0.017s
I20240506 15:15:05.579773 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.592439 21028 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
W20240506 15:15:05.595616 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 73032.1us
I20240506 15:15:05.603683 21032 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:05.608541 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.610272 20950 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:05.613066 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.614360 22939 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:05.614329 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:05.615178 23102 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:05.615985 23102 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:05.616122 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.616592 23102 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:05.617349 22939 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:05.617619 23145 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:05.618554 22939 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.625507 20950 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:05.617712 23102 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.627154 23101 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.627796 23101 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.629796 23101 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.634865 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:05.636734 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:05.639009 20958 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:15:05.651283 23310 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20240506 15:15:05.653435 20996 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.655259 23289 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:05.655881 23289 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:05.656077 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.656301 23289 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.657049 21032 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:05.655174 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.658188 21028 raft_consensus.cc:2354] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:05.661226 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.662856 21032 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:05.664837 23071 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.667097 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:05.667299 21032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:05.667925 20881 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:05.668045 21032 raft_consensus.cc:595] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.668496 21032 raft_consensus.cc:603] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:05.670171 23071 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.670292 20878 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:05.670805 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:05.674263 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:05.676986 20877 raft_consensus.cc:1261] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:05.687345 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:05.688498 21032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:05.690016 21032 raft_consensus.cc:595] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.692198 23098 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.694167 23098 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.696509 23247 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:05.697288 23098 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.703120 23103 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.703812 23103 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.705911 23103 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.706305 21030 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:05.709426 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:05.710249 21028 raft_consensus.cc:595] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.712322 20958 raft_consensus.cc:2454] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:15:05.710546 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 54651.9us
I20240506 15:15:05.717263 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:05.719707 21032 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:05.722966 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:05.724653 20779 catalog_manager.cc:5484] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:05.727505 20952 raft_consensus.cc:1226] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:05.728183 20779 catalog_manager.cc:5484] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:05.730258 22930 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:05.732434 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.732542 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.733451 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.733834 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.734704 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.735006 20950 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.735415 21032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:05.739943 21032 raft_consensus.cc:595] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:05.740604 21032 raft_consensus.cc:603] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:05.699016 23071 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20240506 15:15:05.760465 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.761379 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.766649 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:05.768532 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:05.769305 20952 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.779299 23243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59110 (request call id 210) took 3104 ms. Trace:
I20240506 15:15:05.781697 22939 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:05.782183 22939 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:05.783253 22939 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.786089 22930 consensus_queue.cc:1046] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:05.787552 23250 raft_consensus.cc:979] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:05.788058 23250 raft_consensus.cc:1067] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:05.788136 23226 raft_consensus.cc:3032] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:05.788714 23226 raft_consensus.cc:729] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:05.789721 23226 consensus_queue.cc:260] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.791246 20846 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:05.702462 23289 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.806195 20950 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:05.804818 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:05.810434 20953 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:05.808708 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:05.817013 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 55941.4us
I20240506 15:15:05.818857 20882 raft_consensus.cc:2379] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:05.818899 23101 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:05.830696 23294 consensus_queue.cc:1035] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:05.831810 23202 raft_consensus.cc:2781] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:15:05.826443 20845 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.835855 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:05.815642 23103 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.832487 23202 raft_consensus.cc:686] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:05.861419 23103 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.861941 23202 consensus_queue.cc:237] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:05.864039 23103 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:05.864001 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:05.867887 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 113745us
I20240506 15:15:05.870625 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:05.872608 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:05.876188 21032 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:05.879184 23289 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:05.846632 23247 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.848945 23071 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:05.819332 23098 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240506 15:15:05.885900 20921 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:05.864790 22939 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:05.838891 23299 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240506 15:15:05.895026 22880 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:05.888206 23135 consensus_queue.cc:1046] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20240506 15:15:05.934039 23048 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:05.779857 23243 rpcz_store.cc:276] 0506 15:15:02.675065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:02.762341 (+ 87276us) service_pool.cc:225] Handling call
0506 15:15:03.366954 (+604613us) ts_tablet_manager.cc:903] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
0506 15:15:05.537667 (+2170713us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:15:05.676380 (+138713us) tablet_metadata.cc:641] Metadata flushed
0506 15:15:05.676711 (+ 331us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:15:05.778239 (+101528us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:15:05.778652 (+ 413us) tablet_replica.cc:205] Creating consensus instance
0506 15:15:05.779043 (+ 391us) ts_tablet_manager.cc:958] Replica registered.
0506 15:15:05.779164 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":106462,"tablet-copy.queue_time_us":415391,"thread_start_us":578152,"threads_started":2}
W20240506 15:15:05.977643 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 194718us
I20240506 15:15:05.970103 23103 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:05.969837 23101 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:05.971963 23289 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:05.971985 23145 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:06.017442 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 135638us
W20240506 15:15:06.020265 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 151339us
I20240506 15:15:06.025259 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:06.026746 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:15:06.031502 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 94774.7us
I20240506 15:15:06.041370 20998 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:06.018540 23103 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:06.016557 23243 tablet_copy_client.cc:791] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
W20240506 15:15:06.047550 23298 log.cc:924] Time spent T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.061s user 0.006s sys 0.000s
I20240506 15:15:06.047896 23243 tablet_copy_client.cc:655] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:15:06.048620 22821 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:06.049173 22821 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.050848 20952 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:06.051153 22821 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:06.053270 23319 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240506 15:15:06.054836 23243 tablet_copy_client.cc:531] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:15:06.035928 23289 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.058140 23289 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:06.060953 23318 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20240506 15:15:06.068781 23102 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:06.069456 23102 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.071568 23102 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.040381 23101 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.081142 23101 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.081830 23101 raft_consensus.cc:2741] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20240506 15:15:06.082567 23101 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:06.083180 23101 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:06.040655 23145 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.086349 23243 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:15:06.086890 23145 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:06.088488 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 45952.2us
I20240506 15:15:06.019474 23048 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.094058 23048 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.095351 23103 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:06.096588 23226 raft_consensus.cc:3037] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:06.097633 23103 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:06.101112 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 59853.7us
I20240506 15:15:06.102249 23321 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:06.102706 23321 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:06.103124 20998 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:06.104447 23135 raft_consensus.cc:979] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:06.105005 23135 raft_consensus.cc:2781] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:06.105077 23323 raft_consensus.cc:1067] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:06.105552 23135 raft_consensus.cc:686] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:06.102471 23289 consensus_queue.cc:1046] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240506 15:15:06.106730 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 135159us
I20240506 15:15:06.104053 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:06.109253 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 85216.9us
I20240506 15:15:06.114142 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:06.103132 23321 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:06.123085 23135 consensus_queue.cc:237] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.110998 23101 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.125485 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.126576 23145 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:06.126978 23299 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:06.127599 23299 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.129936 23299 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.134155 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:06.135602 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.136669 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.136904 22880 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:06.137835 23208 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:06.138343 23208 raft_consensus.cc:686] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:06.139405 22821 raft_consensus.cc:3032] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:06.139266 23208 consensus_queue.cc:237] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.139837 22821 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:06.151958 22821 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.125696 23332 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:06.144665 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.154312 20953 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:06.145682 20846 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.123682 23324 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:06.149030 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.136714 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.192377 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.179445 22880 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:06.198487 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.194579 20954 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:06.195308 23324 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:06.200639 23322 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:06.176815 23332 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.203768 22821 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:06.210053 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.211210 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.215682 20879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:06.216409 20879 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:06.216800 20879 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:06.217657 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:06.219110 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:06.220453 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.224989 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.233891 20881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:06.234572 20881 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:06.234932 20881 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:06.231153 23083 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.227205 23321 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.260663 23299 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:06.285387 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.234102 23322 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.274408 23145 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.273680 23083 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:06.239935 23332 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:06.400689 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 374035us
W20240506 15:15:06.347427 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 55287.2us
I20240506 15:15:06.310724 20881 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.370980 20879 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.288739 23299 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.409888 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.413954 20881 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:06.414014 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 193665us
I20240506 15:15:06.311872 22880 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:06.408720 20879 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.421447 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:06.311432 23323 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:06.423977 21024 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:06.425348 23331 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.426117 23331 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.427695 23333 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:06.320305 23324 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.411830 23299 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
W20240506 15:15:06.451648 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 49932.8us
I20240506 15:15:06.463110 20924 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:15:06.464784 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 240055us
I20240506 15:15:06.470520 20921 leader_election.cc:398] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:06.412011 23145 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.473963 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:06.474802 20958 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:06.481601 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:06.482654 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44617.5us
I20240506 15:15:06.457577 23331 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.486490 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.456861 23333 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.487991 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.488229 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:06.492686 20950 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:06.492776 21027 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:06.493245 21027 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:06.503036 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:06.503814 21026 raft_consensus.cc:3032] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:06.506394 23083 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.506974 23330 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:06.507603 23334 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:06.508055 23334 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:06.508441 23334 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:06.508754 23331 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:06.504148 21026 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:06.516100 23145 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.517222 21026 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.518271 21026 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:15:06.518965 23338 log.cc:924] Time spent T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20240506 15:15:06.522161 23332 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.522018 22880 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.513051 21027 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.523473 21027 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:06.502453 23247 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:06.524888 23247 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.526784 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.495245 23333 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:06.538053 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 51671.8us
W20240506 15:15:06.564568 23336 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20240506 15:15:06.565304 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.565966 23332 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:06.569116 20921 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:06.570012 20958 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:06.573956 21026 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:06.454906 23323 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.571862 20950 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
W20240506 15:15:06.580524 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 76051.1us
I20240506 15:15:06.582005 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:06.568228 23337 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.355s user 0.006s sys 0.004s
I20240506 15:15:06.526839 23247 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.583573 23330 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:06.585906 23341 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20240506 15:15:06.594403 23202 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:06.596511 20845 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:06.598057 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:06.598207 23048 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:06.600278 23048 raft_consensus.cc:686] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:06.600041 21028 raft_consensus.cc:1261] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:06.601227 23048 consensus_queue.cc:237] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.603120 23098 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:06.604359 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.608168 21027 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:06.608966 20954 raft_consensus.cc:1261] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:06.609951 21027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:06.610976 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.612841 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.612994 23098 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:06.614338 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 196) took 9838 ms. Trace:
I20240506 15:15:06.580417 23331 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.618817 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:06.615746 21027 raft_consensus.cc:595] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:06.622357 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.624157 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.648577 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.603122 23321 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:06.652135 20953 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:06.651324 20958 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:06.649940 23098 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.678875 20954 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 7: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:06.678707 23322 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:06.685858 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.692006 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:06.688899 20828 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 7610624a16b44896b6aacea1928ab20e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:06.694670 21030 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
W20240506 15:15:06.693599 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 77206.5us
I20240506 15:15:06.699616 20846 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:06.701086 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.708034 20880 raft_consensus.cc:1226] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:06.708397 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.656425 23299 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.718739 20953 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:06.726801 23360 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:06.731451 20874 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:06.746100 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:06.751205 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.614718 20776 rpcz_store.cc:276] 0506 15:14:56.730650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:56.732491 (+ 1841us) service_pool.cc:225] Handling call
0506 15:15:06.568708 (+9836217us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:03.928345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:03.929118 (+ 773us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:03.929167 (+ 49us) write_op.cc:435] Acquired schema lock
0506 15:15:03.929182 (+ 15us) tablet.cc:660] Decoding operations
0506 15:15:03.930275 (+ 1093us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:03.930339 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:03.930369 (+ 30us) tablet.cc:683] Acquiring locks for 10 operations
0506 15:15:03.931276 (+ 907us) tablet.cc:699] Row locks acquired
0506 15:15:03.931298 (+ 22us) write_op.cc:260] PREPARE: finished
0506 15:15:03.931454 (+ 156us) write_op.cc:270] Start()
0506 15:15:03.931582 (+ 128us) write_op.cc:276] Timestamp: P: 1715008503931403 usec, L: 0
0506 15:15:03.931602 (+ 20us) op_driver.cc:331] REPLICATION: starting
0506 15:15:03.969731 (+ 38129us) log.cc:842] Serialized 4029 byte log entry
0506 15:15:04.486173 (+516442us) op_driver.cc:444] REPLICATION: finished
0506 15:15:04.651271 (+165098us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630b3a716e 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f6311d00dd0
0506 15:15:05.425169 (+773898us) write_op.cc:301] APPLY: starting
0506 15:15:05.433881 (+ 8712us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:05.434209 (+ 328us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:05.434243 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:05.435723 (+ 1480us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:05.436070 (+ 347us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0506 15:15:05.436158 (+ 88us) write_op.cc:312] APPLY: finished
0506 15:15:05.458982 (+ 22824us) log.cc:842] Serialized 76 byte log entry
0506 15:15:05.555337 (+ 96355us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:05.606674 (+ 51337us) write_op.cc:454] Released schema lock
0506 15:15:05.607697 (+ 1023us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:05.659929 (+ 52232us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:05.659966 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":919887,"apply.run_cpu_time_us":13936,"apply.run_wall_time_us":244315,"num_ops":10,"prepare.queue_time_us":481247,"prepare.run_cpu_time_us":30378,"prepare.run_wall_time_us":277409,"raft.queue_time_us":368816,"raft.run_cpu_time_us":53962,"raft.run_wall_time_us":489493,"replication_time_us":554469,"spinlock_wait_cycles":372858752,"thread_start_us":1081407,"threads_started":6,"wal-append.queue_time_us":358462,"wal-append.run_cpu_time_us":10292,"wal-append.run_wall_time_us":1132327}]]}
I20240506 15:15:06.756695 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:06.704403 23202 consensus_queue.cc:1035] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240506 15:15:06.702834 23331 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.771890 23348 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:06.772707 23348 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:06.773190 23348 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:06.778030 23299 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.784854 23333 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.875927 23102 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.876827 23102 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.879606 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:06.884315 23102 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.886106 23340 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:06.889765 23357 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:06.896183 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 168998us
I20240506 15:15:06.754163 23103 consensus_queue.cc:1046] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.136s
I20240506 15:15:06.895130 20879 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
W20240506 15:15:06.863516 23329 log.cc:924] Time spent T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20240506 15:15:06.791906 23323 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:06.904809 23357 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.754146 23350 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.138s
I20240506 15:15:06.909052 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:06.906627 23103 raft_consensus.cc:3032] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:06.906322 23334 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.862398 23331 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.913595 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.925076 20881 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:06.925118 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:06.926503 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:06.926920 23351 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.265s user 0.002s sys 0.000s
I20240506 15:15:06.930740 23357 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.933336 23330 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:06.899516 23340 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.934384 20952 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:06.936034 23340 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:06.914526 23103 raft_consensus.cc:729] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:06.937182 20951 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 7: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:06.937319 23103 consensus_queue.cc:260] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.938724 23103 raft_consensus.cc:3037] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:06.939203 23350 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:06.939832 23350 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.942519 23350 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.948454 23334 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:06.923225 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d85358bc4d3f42d897b4c2868c83c8ec, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:06.950356 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d85358bc4d3f42d897b4c2868c83c8ec, TS ecededcd854842d4867fda9fa63a4b66 -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.952085 23360 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:06.952736 23360 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.954999 23360 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:06.957968 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.960230 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.961863 20956 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:06.962848 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:06.968953 20874 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:06.974603 20952 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:06.975358 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.976357 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:06.977274 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.977598 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:06.977151 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:06.977799 23348 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.979033 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 71) took 3914 ms. Trace:
I20240506 15:15:06.980360 20877 raft_consensus.cc:2354] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7.
I20240506 15:15:06.979395 20779 rpcz_store.cc:276] 0506 15:15:03.064729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:03.064903 (+ 174us) service_pool.cc:225] Handling call
0506 15:15:06.978950 (+3914047us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:05.976868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:06.034119 (+ 57251us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:06.042219 (+ 8100us) write_op.cc:435] Acquired schema lock
0506 15:15:06.072381 (+ 30162us) tablet.cc:660] Decoding operations
0506 15:15:06.072883 (+ 502us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:06.072964 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:06.072990 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0506 15:15:06.073378 (+ 388us) tablet.cc:699] Row locks acquired
0506 15:15:06.073392 (+ 14us) write_op.cc:260] PREPARE: finished
0506 15:15:06.073559 (+ 167us) write_op.cc:270] Start()
0506 15:15:06.073684 (+ 125us) write_op.cc:276] Timestamp: P: 1715008506073499 usec, L: 0
0506 15:15:06.073703 (+ 19us) op_driver.cc:331] REPLICATION: starting
0506 15:15:06.074354 (+ 651us) log.cc:842] Serialized 918 byte log entry
0506 15:15:06.388212 (+313858us) op_driver.cc:444] REPLICATION: finished
0506 15:15:06.588559 (+200347us) write_op.cc:301] APPLY: starting
0506 15:15:06.591518 (+ 2959us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:06.615254 (+ 23736us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:06.679993 (+ 64739us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:06.709518 (+ 29525us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:06.709775 (+ 257us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 15:15:06.709843 (+ 68us) write_op.cc:312] APPLY: finished
0506 15:15:06.710128 (+ 285us) log.cc:842] Serialized 28 byte log entry
0506 15:15:06.710339 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:06.710498 (+ 159us) write_op.cc:454] Released schema lock
0506 15:15:06.710567 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:06.710938 (+ 371us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:06.710967 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9837,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":180275,"apply.run_cpu_time_us":2944,"apply.run_wall_time_us":142460,"num_ops":2,"prepare.queue_time_us":218262,"prepare.run_cpu_time_us":3652,"prepare.run_wall_time_us":97805,"replication_time_us":314409,"thread_start_us":152700,"threads_started":2}]]}
I20240506 15:15:06.997720 20951 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:06.998481 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:07.000586 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:07.003702 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 208b7cbf36514c419696de9c14606c16, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.004086 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 208b7cbf36514c419696de9c14606c16, TS 7610624a16b44896b6aacea1928ab20e -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:06.979840 23348 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:07.008654 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.011866 20875 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:07.013913 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.014580 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.015211 20949 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:07.014729 20958 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:07.009385 20951 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:07.020308 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.020926 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.020988 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.036207 21027 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:07.036302 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.038977 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:07.041990 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:07.042790 20950 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.044170 20951 raft_consensus.cc:2379] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:07.043191 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.049371 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.053211 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.062796 20921 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.068861 23344 consensus_queue.cc:1035] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:07.072592 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:07.073222 20951 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:07.074069 20950 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:07.074107 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.076335 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.077324 20921 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.079025 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.070467 22936 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:07.081797 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.080389 23333 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.064837 23330 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.078557 23102 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:07.078480 23101 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.083209 23332 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:07.097695 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.099108 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.104102 21030 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:07.104754 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.106771 23186 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:07.106736 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.111210 20954 raft_consensus.cc:1261] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240506 15:15:07.111639 20954 pending_rounds.cc:76] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 4
I20240506 15:15:07.080479 22936 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.111894 20954 pending_rounds.cc:97] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240506 15:15:07.112411 20954 raft_consensus.cc:2866] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:07.113816 22936 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:07.114364 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.116745 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.117424 21024 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 7
I20240506 15:15:07.117944 23289 raft_consensus.cc:979] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:07.118568 23289 raft_consensus.cc:1067] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:07.086858 23333 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.134608 21024 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:07.140362 20845 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:07.140818 20828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.141178 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 286345f896bc43ec975183896aee147c, TS 00fe71c1675441379f5f2c7490f9e63a -> TS 7c8800c6dcaf4b6e8dd58c4149f73c4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.188367 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.189859 20951 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:07.197727 21026 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:07.198566 21023 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:07.198720 21030 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:07.197512 20845 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:07.199846 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 112493us
I20240506 15:15:07.207118 20845 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:07.190678 23048 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20240506 15:15:07.195611 23367 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20240506 15:15:07.209187 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 116139us
I20240506 15:15:07.197064 22939 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.242568 20950 raft_consensus.cc:1261] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20240506 15:15:07.254894 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 55904.5us
I20240506 15:15:07.254191 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.224040 23098 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
W20240506 15:15:07.260414 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 178679us
I20240506 15:15:07.266526 20921 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.279723 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.284633 20996 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.232527 23048 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20240506 15:15:07.232519 23299 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
W20240506 15:15:07.295776 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 77750.6us
I20240506 15:15:07.232801 23202 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240506 15:15:07.281945 23103 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:07.322671 20921 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.286218 23357 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:07.327206 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:07.330093 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 88745.9us
I20240506 15:15:07.329927 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:07.296676 23289 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:07.275875 23352 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20240506 15:15:07.328598 23204 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:07.356338 22936 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:07.355787 23204 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.359781 23204 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:07.360800 20874 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240506 15:15:07.362077 23377 log.cc:924] Time spent T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20240506 15:15:07.361056 23372 log.cc:924] Time spent T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20240506 15:15:07.329465 23375 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:07.366308 23380 consensus_queue.cc:1046] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240506 15:15:07.329010 23347 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:07.364776 23101 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:07.380645 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 50553.5us
I20240506 15:15:07.354030 22891 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240506 15:15:07.388314 23368 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
I20240506 15:15:07.388617 22936 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:07.389019 22936 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:07.389942 23347 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.396483 22891 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:07.397133 22891 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.387655 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.389955 22936 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.409642 22936 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:07.414811 23243 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:15:07.416064 23243 tablet_bootstrap.cc:492] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:15:07.417130 23243 ts_tablet_manager.cc:1367] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 1.331s user 0.042s sys 0.050s
I20240506 15:15:07.418987 23321 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:07.423064 23379 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:07.423537 23379 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:07.423918 23379 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.392192 23347 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:07.420650 23243 raft_consensus.cc:348] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:07.432804 23226 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240506 15:15:07.433008 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45483.2us
I20240506 15:15:07.435829 23226 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:07.437103 23226 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:07.438373 23331 raft_consensus.cc:3032] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:07.438803 23331 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:07.440568 23250 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:07.440681 20998 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:07.441084 23250 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:07.442301 20998 leader_election.cc:398] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7.
I20240506 15:15:07.441973 23250 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.442606 20998 leader_election.cc:401] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:07.443320 23250 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:07.444518 22930 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:07.445143 22930 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7.
I20240506 15:15:07.439643 23331 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.446530 23331 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.449779 23383 consensus_queue.cc:1046] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:07.450866 23340 raft_consensus.cc:3032] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:07.451256 23340 raft_consensus.cc:729] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:07.451998 23340 consensus_queue.cc:260] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.452529 22880 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:07.454145 23098 raft_consensus.cc:686] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:07.455067 23098 consensus_queue.cc:237] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.465271 20921 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.467000 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.467928 23103 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:07.468605 23145 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:07.470492 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 128382us
I20240506 15:15:07.434883 23243 raft_consensus.cc:729] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:15:07.487505 20874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
W20240506 15:15:07.479107 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 149239us
I20240506 15:15:07.490705 20874 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:07.490826 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:07.491163 20874 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:07.489610 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:07.497469 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:07.504004 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 242911us
I20240506 15:15:07.498034 21027 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:07.509272 20846 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:07.519354 20951 raft_consensus.cc:1226] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20240506 15:15:07.525033 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47625.1us
I20240506 15:15:07.528465 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.533735 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:07.525198 23340 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.542891 23390 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:07.543524 23390 raft_consensus.cc:686] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:07.544541 20845 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:07.544471 23390 consensus_queue.cc:237] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.600946 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 55086.6us
I20240506 15:15:07.605871 20874 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.607692 23048 consensus_queue.cc:1035] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:07.614526 20874 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:07.622112 21025 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:07.611557 23048 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:07.694245 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:07.607519 23379 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.716321 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 114540us
I20240506 15:15:07.529073 23243 consensus_queue.cc:260] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:07.685039 23048 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:07.651923 23145 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20240506 15:15:07.751384 21025 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:07.760046 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:07.760663 20952 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:07.738517 23048 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:07.770334 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.771246 21029 raft_consensus.cc:3032] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:07.771628 21029 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:07.777889 21024 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:07.775127 20778 catalog_manager.cc:5484] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.787485 20952 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:07.783289 20778 catalog_manager.cc:5484] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.795995 20958 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:07.793165 21029 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.797664 20778 catalog_manager.cc:5484] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.747043 23373 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
W20240506 15:15:07.815527 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 61733.6us
I20240506 15:15:07.828841 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:07.832751 20958 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:07.847944 20778 catalog_manager.cc:5484] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 3, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 3 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.850247 20778 catalog_manager.cc:5484] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.847965 22880 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 15:15:07.830793 22891 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.904368 23391 log.cc:924] Time spent T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20240506 15:15:07.905453 23388 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20240506 15:15:07.861092 23098 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20240506 15:15:07.906000 22891 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:07.907766 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:07.908646 20951 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.914932 20954 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:07.889731 22880 raft_consensus.cc:686] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:07.875834 23360 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20240506 15:15:07.913348 20778 catalog_manager.cc:5484] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:07.913475 23379 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:07.931433 20951 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:07.936120 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:07.910856 23385 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240506 15:15:07.941224 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.941920 21024 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:07.942098 23299 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:07.942608 23299 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
W20240506 15:15:07.952149 23397 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20240506 15:15:07.943483 23299 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:07.953784 23396 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20240506 15:15:07.960772 22880 consensus_queue.cc:237] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.931350 23350 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20240506 15:15:07.963934 23398 log.cc:924] Time spent T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20240506 15:15:07.966720 23395 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20240506 15:15:07.970855 21024 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:07.973677 23243 ts_tablet_manager.cc:1398] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.556s user 0.010s sys 0.000s
W20240506 15:15:07.974063 23243 ts_tablet_manager.cc:1430] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Tablet startup took 556ms
W20240506 15:15:07.978543 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 150257us
I20240506 15:15:07.983465 23048 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:07.985872 23048 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:07.989076 20778 catalog_manager.cc:5484] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:07.990566 20879 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240506 15:15:07.991020 20778 catalog_manager.cc:5484] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 4, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:07.991525 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:07.992055 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:07.992686 21031 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:08.010226 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:08.020301 20996 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.018105 23145 consensus_queue.cc:1046] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240506 15:15:08.041610 23145 raft_consensus.cc:3032] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:08.041426 23407 consensus_queue.cc:1035] T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:08.041996 23145 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:15:08.042896 23405 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20240506 15:15:08.043079 23250 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:07.974375 23243 ts_tablet_manager.cc:1432] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Trace:
0506 15:15:02.675065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:02.762341 (+ 87276us) service_pool.cc:225] Handling call
0506 15:15:03.366954 (+604613us) ts_tablet_manager.cc:903] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
0506 15:15:05.537667 (+2170713us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:15:05.676380 (+138713us) tablet_metadata.cc:641] Metadata flushed
0506 15:15:05.676711 (+ 331us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:15:05.778239 (+101528us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:15:05.778652 (+ 413us) tablet_replica.cc:205] Creating consensus instance
0506 15:15:05.779043 (+ 391us) ts_tablet_manager.cc:958] Replica registered.
0506 15:15:05.779164 (+ 121us) inbound_call.cc:173] Queueing success response
0506 15:15:06.085984 (+306820us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0506 15:15:07.417553 (+1331569us) ts_tablet_manager.cc:1399] Starting tablet replica
0506 15:15:07.418124 (+ 571us) tablet_replica.cc:259] Starting instrumentation
0506 15:15:07.418830 (+ 706us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":106462,"mutex_wait_us":20710,"raft.queue_time_us":27559,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":7872,"tablet-copy.queue_time_us":415391,"tablet-download-pool.queue_time_us":236,"tablet-download-pool.run_cpu_time_us":1147,"tablet-download-pool.run_wall_time_us":6023,"thread_start_us":578152,"threads_started":2}
I20240506 15:15:08.048626 21031 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:08.042934 23145 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.053079 20996 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.053207 23145 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:08.063772 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:08.054031 23409 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:08.056159 20950 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:08.083647 21031 raft_consensus.cc:1261] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:08.085896 20845 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.091799 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-b540a7832fba49b5b2623616240c2711 received from {username='slave'} at 127.0.0.1:49620
I20240506 15:15:08.092358 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-b540a7832fba49b5b2623616240c2711 on tablet b540a7832fba49b5b2623616240c2711 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:08.093734 23360 consensus_queue.cc:1046] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240506 15:15:08.132242 23250 consensus_queue.cc:1159] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ecededcd854842d4867fda9fa63a4b66" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20240506 15:15:08.090346 23071 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:08.138079 23071 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:08.139077 23071 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.148800 23360 raft_consensus.cc:3032] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:08.152730 23360 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:08.153553 23360 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.154940 23360 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:08.166254 20996 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.189327 22930 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:08.204903 23344 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:08.254082 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 45307.6us
I20240506 15:15:08.233357 23346 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:08.262761 23416 log.cc:924] Time spent T 3ca08769950c4ef2bae0df854c025154 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20240506 15:15:08.264639 21029 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:08.282305 23343 consensus_queue.cc:1035] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:08.263202 23346 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:08.286520 23346 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.287353 22939 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:08.237625 23344 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.290285 23344 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.293311 20955 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:08.300760 23330 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:08.301259 23346 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:08.301592 23330 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.303982 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:08.313958 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 90784.5us
W20240506 15:15:08.316185 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 92963.9us
I20240506 15:15:08.316923 23330 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.319703 21029 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:08.324710 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.325521 21029 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:08.326799 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:08.337589 23103 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:08.337652 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:08.339169 20878 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:08.341150 21689 raft_consensus.cc:1203] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240506 15:15:08.343542 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:08.358137 20874 raft_consensus.cc:1226] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:08.338212 23103 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:08.361963 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 88332.8us
I20240506 15:15:08.391299 21030 raft_consensus.cc:1261] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240506 15:15:08.391662 21030 pending_rounds.cc:76] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Aborting all ops after (but not including) 4
I20240506 15:15:08.365590 23103 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:08.391891 21030 pending_rounds.cc:97] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240506 15:15:08.393728 21030 raft_consensus.cc:2866] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:08.397492 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.391100 23299 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:08.412758 23299 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.417443 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:08.418373 20950 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:08.405475 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.422854 23299 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.445648 23346 consensus_queue.cc:1035] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:08.451839 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.458061 20845 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.463258 23299 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:08.465543 23344 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240506 15:15:08.537377 23426 log.cc:924] Time spent T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20240506 15:15:08.605444 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240506 15:15:08.610247 23428 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.118s user 0.000s sys 0.003s
I20240506 15:15:08.610452 23103 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:08.631543 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 163968us
I20240506 15:15:08.636458 23103 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:08.641834 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 174659us
I20240506 15:15:08.646940 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.647636 21029 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:08.649228 21029 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:08.678303 21029 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:08.674122 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.681035 21031 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:08.688625 20949 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:08.697997 23334 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:08.698472 23334 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:08.698865 23334 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:08.705753 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:08.720335 20882 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:08.723632 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:08.724673 23373 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:08.730868 20955 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240506 15:15:08.724820 23103 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:08.731786 23103 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.731153 23373 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.736354 23373 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.737313 23103 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.738061 21029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:08.741053 21029 raft_consensus.cc:595] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.741405 21029 raft_consensus.cc:603] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:08.742708 23299 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:08.743984 23299 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:08.726851 22891 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20240506 15:15:08.745515 22891 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:08.748445 23208 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:08.751375 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:08.751608 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:08.759639 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.760300 21029 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:08.761183 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.761828 21025 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:08.784340 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:08.823129 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:08.826434 20958 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240506 15:15:08.826854 20958 pending_rounds.cc:76] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 3
I20240506 15:15:08.827059 20958 pending_rounds.cc:97] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240506 15:15:08.827303 20958 raft_consensus.cc:2866] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:08.827302 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:08.829241 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:08.838809 21029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:08.839654 21029 raft_consensus.cc:595] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.840119 21029 raft_consensus.cc:603] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:08.832170 23331 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:08.847523 23331 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:08.848088 23331 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:08.845042 23289 raft_consensus.cc:1050] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:08.833575 23299 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20240506 15:15:08.831988 23101 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:08.852782 23101 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:08.853319 23101 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:08.844740 23048 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:08.855051 23048 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.857125 23048 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.840962 23145 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:08.867677 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:08.870777 21027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:08.871462 21027 raft_consensus.cc:595] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.871902 21027 raft_consensus.cc:603] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:08.876641 23145 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:08.878119 23145 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.881111 23334 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.883177 23334 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:08.884416 23436 log.cc:924] Time spent T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20240506 15:15:08.886814 20876 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:08.889932 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:08.895637 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:08.896420 21027 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
W20240506 15:15:08.899173 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 97505.6us
I20240506 15:15:08.908700 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:08.916960 20845 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:08.921052 23204 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:08.921562 23204 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:08.922044 23204 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:08.918071 23289 consensus_queue.cc:237] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:08.926254 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:08.926992 21028 raft_consensus.cc:595] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.927306 21028 raft_consensus.cc:603] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:08.928716 21029 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:08.937265 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:08.946524 23331 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.950467 23331 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:08.951154 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:08.952596 23101 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.944720 23344 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:08.967799 23344 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:08.969283 23344 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:08.960093 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:08.971846 21027 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:08.947935 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:08.976733 21028 raft_consensus.cc:595] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:08.977243 21028 raft_consensus.cc:603] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:08.983575 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:08.985605 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:08.986812 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:09.005272 21027 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:09.015547 20921 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:09.026094 20953 raft_consensus.cc:2354] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:09.022929 23331 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:09.055229 23331 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:09.056171 23331 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.074513 23101 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:09.081041 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:09.081738 21031 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:09.091120 21031 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:09.111274 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:09.111472 20776 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.113586 20776 catalog_manager.cc:5484] T 7e60758583b247c2946eb30ea2681286 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 5 to 7, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 7 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.115684 20776 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 5 to 6, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.126948 20846 leader_election.cc:398] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:09.131817 23448 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:09.132297 21689 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:15:09.132779 23448 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:09.133919 23448 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.172102 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:09.173686 23322 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:09.174892 23204 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.176931 23204 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:09.178119 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:09.178737 21031 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:09.181856 20951 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240506 15:15:09.184950 23323 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:09.202613 20876 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20240506 15:15:09.205159 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 50673.7us
I20240506 15:15:09.205531 23322 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:09.221604 23204 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:09.217890 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:09.231436 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:09.235102 20845 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:09.229192 23204 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.245071 23204 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:09.236018 23299 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:09.249228 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:09.254354 21031 raft_consensus.cc:2354] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9.
I20240506 15:15:09.258052 20845 leader_election.cc:398] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9.
I20240506 15:15:09.258484 20845 leader_election.cc:401] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:09.261477 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:09.260618 23299 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
W20240506 15:15:09.219525 20778 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 176) took 13687 ms (client timeout 15000 ms). Trace:
I20240506 15:15:09.269074 23423 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 9
I20240506 15:15:09.271394 23331 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:09.283164 23331 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.285275 23331 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:09.276476 20778 rpcz_store.cc:269] 0506 15:14:55.544862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:14:55.557937 (+ 13075us) service_pool.cc:225] Handling call
0506 15:14:55.778825 (+220888us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b54000037d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e274d30 00007f630e2f817a 00007f6314249b1c 00007f6314343a8e 00007f6312657a44 00007f63126579d2 00007f6312657901 00007f631265787c 00007f6312656b32 00007f630c716bcc 00007f630c716006
0506 15:15:09.219442 (+13440617us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:08.295098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:08.296912 (+ 1814us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:08.296945 (+ 33us) write_op.cc:435] Acquired schema lock
0506 15:15:08.296965 (+ 20us) tablet.cc:660] Decoding operations
0506 15:15:08.302760 (+ 5795us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:08.302840 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:08.302867 (+ 27us) tablet.cc:683] Acquiring locks for 8 operations
0506 15:15:08.303866 (+ 999us) tablet.cc:699] Row locks acquired
0506 15:15:08.303893 (+ 27us) write_op.cc:260] PREPARE: finished
0506 15:15:08.304054 (+ 161us) write_op.cc:270] Start()
0506 15:15:08.304154 (+ 100us) write_op.cc:276] Timestamp: P: 1715008508304002 usec, L: 0
0506 15:15:08.304175 (+ 21us) op_driver.cc:331] REPLICATION: starting
0506 15:15:08.317937 (+ 13762us) log.cc:842] Serialized 3257 byte log entry
0506 15:15:08.803259 (+485322us) op_driver.cc:444] REPLICATION: finished
0506 15:15:08.854652 (+ 51393us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630b3a716e 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f6311d00dd0
0506 15:15:09.046760 (+192108us) write_op.cc:301] APPLY: starting
0506 15:15:09.051657 (+ 4897us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:09.055967 (+ 4310us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:09.055993 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:09.057148 (+ 1155us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:09.057345 (+ 197us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0506 15:15:09.057387 (+ 42us) write_op.cc:312] APPLY: finished
0506 15:15:09.057628 (+ 241us) log.cc:842] Serialized 64 byte log entry
0506 15:15:09.057811 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.057926 (+ 115us) write_op.cc:454] Released schema lock
0506 15:15:09.057967 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:09.058226 (+ 259us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.058255 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":47,"spinlock_wait_cycles":3482880,"child_traces":[["op",{"apply.queue_time_us":222500,"apply.run_cpu_time_us":3086,"apply.run_wall_time_us":29214,"num_ops":8,"prepare.queue_time_us":221431,"prepare.run_cpu_time_us":15093,"prepare.run_wall_time_us":100239,"raft.queue_time_us":191555,"raft.run_cpu_time_us":20563,"raft.run_wall_time_us":154037,"replication_time_us":498988,"spinlock_wait_cycles":21413632,"thread_start_us":523388,"threads_started":4,"wal-append.queue_time_us":136613}]]}
I20240506 15:15:09.288893 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:09.289635 21029 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:09.289381 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:09.291823 20882 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:09.308069 23299 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.328047 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:09.322319 23423 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9.
I20240506 15:15:09.338678 23448 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:09.348481 21029 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:09.340917 23448 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:09.352103 23101 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240506 15:15:09.359613 23447 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:09.360242 23447 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:09.359676 20878 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.361085 23447 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.362350 23447 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:09.372854 21025 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:09.355465 23448 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:09.382860 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 72) took 2394 ms. Trace:
I20240506 15:15:09.393702 23346 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:09.369781 23250 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.402235 23346 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:09.383327 20776 rpcz_store.cc:276] 0506 15:15:06.988587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:06.989439 (+ 852us) service_pool.cc:225] Handling call
0506 15:15:09.382790 (+2393351us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:09.310216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:09.310752 (+ 536us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:09.310800 (+ 48us) write_op.cc:435] Acquired schema lock
0506 15:15:09.310819 (+ 19us) tablet.cc:660] Decoding operations
0506 15:15:09.311383 (+ 564us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:09.311456 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:09.311483 (+ 27us) tablet.cc:683] Acquiring locks for 3 operations
0506 15:15:09.311817 (+ 334us) tablet.cc:699] Row locks acquired
0506 15:15:09.311837 (+ 20us) write_op.cc:260] PREPARE: finished
0506 15:15:09.311975 (+ 138us) write_op.cc:270] Start()
0506 15:15:09.312083 (+ 108us) write_op.cc:276] Timestamp: P: 1715008509311933 usec, L: 0
0506 15:15:09.312104 (+ 21us) op_driver.cc:331] REPLICATION: starting
0506 15:15:09.312799 (+ 695us) log.cc:842] Serialized 1301 byte log entry
0506 15:15:09.316544 (+ 3745us) op_driver.cc:444] REPLICATION: finished
0506 15:15:09.316893 (+ 349us) write_op.cc:301] APPLY: starting
0506 15:15:09.316975 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:09.317120 (+ 145us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:09.317215 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:09.317629 (+ 414us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:09.317789 (+ 160us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0506 15:15:09.317849 (+ 60us) write_op.cc:312] APPLY: finished
0506 15:15:09.327972 (+ 10123us) log.cc:842] Serialized 34 byte log entry
0506 15:15:09.336810 (+ 8838us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.339588 (+ 2778us) write_op.cc:454] Released schema lock
0506 15:15:09.340738 (+ 1150us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:09.348556 (+ 7818us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.350703 (+ 2147us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":46332,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":114,"apply.run_cpu_time_us":2473,"apply.run_wall_time_us":35505,"num_ops":3,"prepare.queue_time_us":159751,"prepare.run_cpu_time_us":2880,"prepare.run_wall_time_us":2879,"replication_time_us":4022,"thread_start_us":130469,"threads_started":1}]]}
I20240506 15:15:09.402666 23346 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:09.403486 23346 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:09.416163 23455 log.cc:924] Time spent T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20240506 15:15:09.418576 23346 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:09.432703 20779 catalog_manager.cc:5484] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.435508 20779 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.438722 20951 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.443397 21689 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.492111 20779 catalog_manager.cc:5484] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 6, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.495383 20779 catalog_manager.cc:5484] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.509009 23448 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.556551 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:09.559434 23448 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:09.555037 20779 catalog_manager.cc:5484] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.562075 20779 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.563885 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:09.576825 21023 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:09.569626 20878 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:09.578429 20924 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:09.580615 23448 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:09.580937 20779 catalog_manager.cc:5484] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.582332 20779 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 8, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 8 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.583847 20779 catalog_manager.cc:5484] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 1 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.585399 20779 catalog_manager.cc:5484] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.586671 20779 catalog_manager.cc:5484] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.581398 23448 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:09.596660 20881 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240506 15:15:09.601761 23447 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:09.587944 20779 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.613147 20779 catalog_manager.cc:5484] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 6, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.614605 20779 catalog_manager.cc:5484] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:09.627048 23448 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:09.628314 20779 catalog_manager.cc:5484] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.634267 20779 catalog_manager.cc:5484] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.636662 20779 catalog_manager.cc:5484] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:09.639298 20779 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.657182 21025 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240506 15:15:09.665570 23447 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:09.667070 20779 catalog_manager.cc:5484] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:09.670670 20779 catalog_manager.cc:5484] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 6, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:09.685854 20779 catalog_manager.cc:5484] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 7, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.689421 20779 catalog_manager.cc:5484] T d85358bc4d3f42d897b4c2868c83c8ec P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.691278 20779 catalog_manager.cc:5484] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.707360 20955 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20240506 15:15:09.707809 20955 pending_rounds.cc:76] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 5
I20240506 15:15:09.708025 20955 pending_rounds.cc:97] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20240506 15:15:09.708237 20955 raft_consensus.cc:2866] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:09.708305 20779 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.709636 20779 catalog_manager.cc:5484] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 5, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.711094 20779 catalog_manager.cc:5484] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.718730 23394 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:09.734354 21023 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20240506 15:15:09.719935 20779 catalog_manager.cc:5484] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 4 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.740653 23447 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:09.743245 20779 catalog_manager.cc:5484] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 6 to 7, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.759055 21023 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240506 15:15:09.761921 23346 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:09.764461 20881 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20240506 15:15:09.764917 20881 pending_rounds.cc:76] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 2
I20240506 15:15:09.752107 20779 catalog_manager.cc:5484] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.765161 20881 pending_rounds.cc:97] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20240506 15:15:09.767154 20881 raft_consensus.cc:2866] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:09.768292 20779 catalog_manager.cc:5484] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 3 to 7, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:09.774977 20779 catalog_manager.cc:5484] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:09.776674 20779 catalog_manager.cc:5484] T 91da71d5ebf84ae7844608806c0d9a94 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 8, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 8 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:09.826690 23145 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:09.880017 20776 catalog_manager.cc:5484] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 4, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:09.899248 20780 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:09.935707 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 197) took 1321 ms. Trace:
I20240506 15:15:09.936213 20779 rpcz_store.cc:276] 0506 15:15:08.614197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:08.615424 (+ 1227us) service_pool.cc:225] Handling call
0506 15:15:09.803706 (+1188282us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.803737 (+ 31us) write_op.cc:454] Released schema lock
0506 15:15:09.935593 (+131856us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:09.787894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:09.788374 (+ 480us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:09.788402 (+ 28us) write_op.cc:435] Acquired schema lock
0506 15:15:09.788415 (+ 13us) tablet.cc:660] Decoding operations
0506 15:15:09.790613 (+ 2198us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:09.790669 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:09.790701 (+ 32us) tablet.cc:683] Acquiring locks for 32 operations
0506 15:15:09.792929 (+ 2228us) tablet.cc:699] Row locks acquired
0506 15:15:09.792955 (+ 26us) write_op.cc:260] PREPARE: finished
0506 15:15:09.793097 (+ 142us) write_op.cc:270] Start()
0506 15:15:09.793199 (+ 102us) write_op.cc:276] Timestamp: P: 1715008509793051 usec, L: 0
0506 15:15:09.793219 (+ 20us) op_driver.cc:331] REPLICATION: starting
0506 15:15:09.793837 (+ 618us) log.cc:842] Serialized 12623 byte log entry
0506 15:15:09.797342 (+ 3505us) op_driver.cc:444] REPLICATION: finished
0506 15:15:09.797814 (+ 472us) write_op.cc:301] APPLY: starting
0506 15:15:09.797890 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:09.798438 (+ 548us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:09.798532 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:09.801068 (+ 2536us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:09.801435 (+ 367us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=32
0506 15:15:09.801479 (+ 44us) write_op.cc:312] APPLY: finished
0506 15:15:09.802014 (+ 535us) log.cc:842] Serialized 211 byte log entry
0506 15:15:09.802192 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.802486 (+ 294us) write_op.cc:454] Released schema lock
0506 15:15:09.802565 (+ 79us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":11492,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":252,"apply.run_cpu_time_us":5020,"apply.run_wall_time_us":5020,"num_ops":32,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":6219,"prepare.run_wall_time_us":15714,"replication_time_us":4017}]]}
I20240506 15:15:09.987960 20778 catalog_manager.cc:5484] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.002660 20778 catalog_manager.cc:5484] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 3 to 4, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 4 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:10.004130 20778 catalog_manager.cc:5484] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.030077 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54786 (request call id 193) took 4832 ms. Trace:
I20240506 15:15:10.031934 21029 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:10.037566 23145 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:10.033283 20780 rpcz_store.cc:276] 0506 15:15:05.197603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:05.198015 (+ 412us) service_pool.cc:225] Handling call
0506 15:15:10.029974 (+4831959us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:09.921374 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:09.921862 (+ 488us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:09.921891 (+ 29us) write_op.cc:435] Acquired schema lock
0506 15:15:09.921907 (+ 16us) tablet.cc:660] Decoding operations
0506 15:15:09.922208 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:09.922260 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:09.922287 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:15:09.922464 (+ 177us) tablet.cc:699] Row locks acquired
0506 15:15:09.922484 (+ 20us) write_op.cc:260] PREPARE: finished
0506 15:15:09.922602 (+ 118us) write_op.cc:270] Start()
0506 15:15:09.922706 (+ 104us) write_op.cc:276] Timestamp: P: 1715008509922561 usec, L: 0
0506 15:15:09.922728 (+ 22us) op_driver.cc:331] REPLICATION: starting
0506 15:15:09.923514 (+ 786us) log.cc:842] Serialized 540 byte log entry
0506 15:15:09.925321 (+ 1807us) op_driver.cc:444] REPLICATION: finished
0506 15:15:09.925726 (+ 405us) write_op.cc:301] APPLY: starting
0506 15:15:09.925805 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:09.925896 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:09.926902 (+ 1006us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:09.927646 (+ 744us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:09.927795 (+ 149us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:15:09.927849 (+ 54us) write_op.cc:312] APPLY: finished
0506 15:15:09.928034 (+ 185us) log.cc:842] Serialized 22 byte log entry
0506 15:15:09.928182 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.928281 (+ 99us) write_op.cc:454] Released schema lock
0506 15:15:09.928325 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:09.928583 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:09.928616 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":58808,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":1705,"apply.run_wall_time_us":3082,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2327,"prepare.run_wall_time_us":2412,"replication_time_us":2505}]]}
I20240506 15:15:10.058529 20779 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 8 to 10, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 10 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.069588 20881 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:10.079969 23331 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:10.112468 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:10.277784 20779 catalog_manager.cc:5484] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 8, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 8 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:10.434054 21029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:10.435328 21029 raft_consensus.cc:595] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.526587 21025 consensus_queue.cc:237] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:10.549527 20882 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:10.550194 20882 raft_consensus.cc:595] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.557343 20882 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:10.558123 20882 raft_consensus.cc:595] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.560771 20882 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:10.561578 20882 raft_consensus.cc:595] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.564834 20878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:10.565644 20878 raft_consensus.cc:595] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.571314 20878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:10.571923 20878 raft_consensus.cc:595] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:10.573374 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
W20240506 15:15:10.573563 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 49866.3us
I20240506 15:15:10.575155 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:10.578838 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:10.619022 20958 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:15:10.623060 21689 raft_consensus.cc:1261] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 15:15:10.621841 23323 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20240506 15:15:10.644081 23250 consensus_queue.cc:1035] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240506 15:15:10.647650 23470 log.cc:924] Time spent T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240506 15:15:10.679165 23394 raft_consensus.cc:979] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.679749 23473 raft_consensus.cc:1067] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.683593 20951 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:10.683986 23250 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:10.709439 20950 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:10.710031 20950 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:10.710368 20950 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.711642 20780 catalog_manager.cc:5484] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New cstate: current_term: 3 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:10.719792 20950 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.759086 20950 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:10.765882 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:10.766598 21029 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.768906 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet b540a7832fba49b5b2623616240c2711 with delete_type TABLET_DATA_TOMBSTONED (TS 00fe71c1675441379f5f2c7490f9e63a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50460
I20240506 15:15:10.773559 21029 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:10.775393 20921 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:10.781538 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:10.782187 20882 raft_consensus.cc:3032] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:10.782835 20882 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:10.783919 23145 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:10.785379 20882 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.787901 20764 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b540a7832fba49b5b2623616240c2711 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240506 15:15:10.788048 20882 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.789237 23344 raft_consensus.cc:979] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.789701 23344 raft_consensus.cc:1067] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.789268 23145 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:10.791741 23145 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.793643 23323 raft_consensus.cc:979] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.794112 23250 raft_consensus.cc:1067] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.803124 23476 tablet_replica.cc:330] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: stopping tablet replica
I20240506 15:15:10.803689 20882 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:10.804520 20951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35810
I20240506 15:15:10.804797 23476 raft_consensus.cc:2229] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240506 15:15:10.805119 20951 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:10.805662 20951 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.808216 23476 raft_consensus.cc:2258] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240506 15:15:10.812233 21689 raft_consensus.cc:2932] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:10.813422 20951 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.815321 20951 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:10.816951 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:10.817728 21023 raft_consensus.cc:3032] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:10.818107 21023 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:10.818794 21023 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.820030 21023 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.825752 21025 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:10.828004 20958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:10.828631 20958 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:10.828992 20958 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.830654 21023 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:10.817099 23476 ts_tablet_manager.cc:1875] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 15:15:10.835628 20958 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.833786 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:10.837064 23447 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:10.837575 23447 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:10.838454 23447 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.837575 20958 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:10.845686 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:10.846477 21031 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.851014 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:10.851689 20878 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.854068 21031 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:10.860057 20921 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:10.862376 23448 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:10.862983 23448 raft_consensus.cc:686] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:10.864163 23448 consensus_queue.cc:237] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.865862 20878 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:10.868520 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:10.869113 20878 raft_consensus.cc:3032] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:10.869413 20878 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:10.882900 20780 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.888450 23476 ts_tablet_manager.cc:1888] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240506 15:15:10.889637 20878 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.890260 23476 log.cc:1196] T b540a7832fba49b5b2623616240c2711 P 00fe71c1675441379f5f2c7490f9e63a: Deleting WAL directory at /tmp/dist-test-task1mugt0/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715008381158361-19726-0/minicluster-data/ts-0-root/wals/b540a7832fba49b5b2623616240c2711
I20240506 15:15:10.890877 20878 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:10.892062 20767 catalog_manager.cc:4830] TS 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): tablet b540a7832fba49b5b2623616240c2711 (table test-workload [id=fdad29afeb4b4c3bb2202b6880d21daf]) successfully deleted
I20240506 15:15:10.908931 23346 raft_consensus.cc:979] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.909560 23344 raft_consensus.cc:1067] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.912812 20779 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 6, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.916375 20950 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:10.916957 20950 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:10.917337 20950 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:10.919667 20878 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:10.923626 20950 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.914477 20779 catalog_manager.cc:5484] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:10.941689 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:10.942313 21023 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:10.945320 20950 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:10.962904 23346 raft_consensus.cc:979] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.963459 23346 raft_consensus.cc:1067] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:10.964434 21023 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:10.965765 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:10.966639 23448 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:10.967015 20950 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:10.967200 23448 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:10.967599 20950 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:10.967963 20950 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.968201 23448 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.974731 20950 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.976192 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:10.976914 20882 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:10.977244 20882 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:10.978022 20882 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.980451 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:10.981457 20882 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:10.981303 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:10.981380 20881 raft_consensus.cc:3032] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:10.982363 21023 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.982509 20881 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:10.983378 20881 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.984690 20881 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:10.989929 21023 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:10.990958 20881 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:10.991943 20924 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:10.992848 23145 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:10.993358 23145 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:10.994240 23145 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:10.996861 20950 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:11.015818 20882 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:11.023998 20779 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 8 to 9, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 9 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:11.047106 20779 catalog_manager.cc:5484] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 6, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:11.048137 23344 raft_consensus.cc:979] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:11.048709 23344 raft_consensus.cc:1067] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:11.052433 20951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:11.053339 20951 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:11.053742 20951 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:11.061841 20951 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:11.080031 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:11.085121 20882 raft_consensus.cc:3032] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:11.090531 20882 raft_consensus.cc:729] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:11.091522 20882 consensus_queue.cc:260] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:11.094034 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:11.094724 21689 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:11.093847 20951 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:11.096385 20882 raft_consensus.cc:3037] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:11.107348 21689 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:11.109010 20922 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:11.109931 23448 raft_consensus.cc:2781] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:11.110445 23448 raft_consensus.cc:686] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:11.111372 23448 consensus_queue.cc:237] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:11.159870 20882 raft_consensus.cc:2454] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:11.162503 20779 catalog_manager.cc:5484] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 6 to 7, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:11.191200 21023 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:11.194739 23448 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:11.244954 21025 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:11.249322 23447 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:11.267150 20878 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:11.269078 20882 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:11.273239 23447 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:11.274012 20882 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:11.274618 23447 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:11.284621 23447 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:11.321594 21031 raft_consensus.cc:1261] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:11.323079 23487 consensus_queue.cc:1035] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:11.380976 20882 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240506 15:15:11.384629 23487 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:15:11.417430 20876 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:11.427814 23487 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:11.431947 21023 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240506 15:15:11.433765 23331 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:11.452441 21031 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:11.454629 23331 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:11.531227 21689 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240506 15:15:11.535447 23331 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240506 15:15:11.569792 20878 raft_consensus.cc:1261] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240506 15:15:11.575342 23145 consensus_queue.cc:1035] T d84d513d60b64cba9c6bc3f8077e6b73 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240506 15:15:11.907647 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240506 15:15:12.045373 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 74615.2us
W20240506 15:15:12.055099 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 176247us
I20240506 15:15:12.078745 20952 consensus_queue.cc:237] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
W20240506 15:15:12.376240 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 288732us
I20240506 15:15:12.557009 21023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:12.557690 21023 raft_consensus.cc:595] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:14.079279 21053 debug-util.cc:398] Leaking SignalData structure 0x7b0800139ce0 after lost signal to thread 19729
W20240506 15:15:14.080876 21053 debug-util.cc:398] Leaking SignalData structure 0x7b080005cd60 after lost signal to thread 20814
W20240506 15:15:14.082221 21053 debug-util.cc:398] Leaking SignalData structure 0x7b08001bae60 after lost signal to thread 20907
W20240506 15:15:14.114050 19731 debug-util.cc:398] Leaking SignalData structure 0x7b080000eb40 after lost signal to thread 20921
W20240506 15:15:14.114573 19731 kernel_stack_watchdog.cc:198] Thread 20921 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 265ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20240506 15:15:14.121019 23530 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.123056 23530 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.124804 23532 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.125305 23530 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.125610 23532 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.126951 23530 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.127768 23532 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.127504 23530 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.129602 23530 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.130666 23532 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.130764 23530 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.131243 23532 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.131366 23530 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.133333 23532 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.133374 23530 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.133970 23532 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.136021 23530 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.136689 23532 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.137004 23530 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.138862 23530 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.139417 23530 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.135797 23533 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.141167 23533 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.150602 23534 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.151481 23534 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.154623 23534 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.157212 23533 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.158115 23532 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.162869 23533 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.163405 23533 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.164237 23532 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.140725 23530 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.164933 23532 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.161849 23535 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.165442 23533 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.168391 23530 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:14.175520 21053 debug-util.cc:398] Leaking SignalData structure 0x7b080039a600 after lost signal to thread 20923
W20240506 15:15:14.177026 21053 debug-util.cc:398] Leaking SignalData structure 0x7b0800265cc0 after lost signal to thread 20982
W20240506 15:15:14.178193 21053 debug-util.cc:398] Leaking SignalData structure 0x7b0800304160 after lost signal to thread 21056
I20240506 15:15:14.158970 23536 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.167095 23534 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.185200 23536 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.193862 23537 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.169301 23535 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.194578 23533 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.194509 23537 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.195221 23530 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.199569 23533 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.200294 23530 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.200781 23532 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.201761 23533 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.203576 23535 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.204121 23535 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.202435 23530 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.209172 23532 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.209232 23533 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.194756 23534 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.214859 23471 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.217240 23471 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.218420 23534 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.214725 23533 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:14.215669 20848 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.602s user 0.033s sys 0.090s
W20240506 15:15:14.220111 20848 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.615s user 0.034s sys 0.090s
W20240506 15:15:14.225881 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 2.13969e+06us
I20240506 15:15:14.227119 23533 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.225868 23471 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.229049 23536 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.205543 23535 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.257437 23534 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.213258 23538 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.254810 23530 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.257238 23533 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.261922 23536 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.220283 23532 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.262100 23531 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.233167 23537 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.284453 23471 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.263304 23534 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.293325 23543 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.293444 23533 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.285773 23540 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.290414 23538 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.304812 23533 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.302114 23546 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.302682 23548 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.302747 23537 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.308794 23545 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.301963 23471 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:14.331177 20847 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.728s user 0.012s sys 0.028s
W20240506 15:15:14.331562 20847 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.729s user 0.013s sys 0.028s
I20240506 15:15:14.297258 23531 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.331854 23471 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.322638 23535 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.329424 23537 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.334712 23531 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.327615 23533 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.334853 23537 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.330391 23545 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.335542 23531 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.335568 23533 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.364548 23531 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.365842 23533 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.366710 23531 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.337818 23534 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.291296 23530 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.339375 23532 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.325138 23540 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.341085 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:14.378927 23530 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:14.382736 20845 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.762s user 0.010s sys 0.005s
W20240506 15:15:14.383081 20845 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.763s user 0.010s sys 0.005s
I20240506 15:15:14.343868 23538 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.345041 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:14.348474 23547 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.349298 23529 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.328244 23548 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:14.386427 23505 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.387195 23505 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.295740 23536 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.353330 23344 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.302817 23543 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.358616 23546 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.360188 23525 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.361181 23513 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.363500 20881 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:14.337616 23545 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.375221 23533 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.377569 23520 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:14.377837 23534 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.377049 23535 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:14.377049 23394 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:14.378891 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:14.381505 23540 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.377094 23531 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.398811 23525 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.392136 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:14.391904 23344 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.410425 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:14.395422 23536 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.399658 23513 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.129280 23568 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.129827 23568 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.130833 23513 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.388242 23547 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.132403 23568 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.391086 23538 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.133373 23547 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.133239 23538 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.135219 23538 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.398434 20881 raft_consensus.cc:595] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:14.387987 23548 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.398877 23553 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.138860 23553 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.140434 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.141607 23553 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.390923 23529 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.148550 23145 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.149133 23529 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.149096 23145 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.149983 21024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:15.150544 21024 raft_consensus.cc:595] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:15.151175 23145 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.400682 23551 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.425660 23552 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.152033 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240506 15:15:15.152314 23552 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.152164 23551 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.408636 23545 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:15.154349 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 3.08224e+06us
I20240506 15:15:15.154641 23545 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.155160 23552 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.155531 23551 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.413928 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.156591 23545 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.414248 23543 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.424237 23546 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.403659 23535 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.160902 23535 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.631191 23556 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.161053 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.161619 23556 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:15.163138 21053 debug-util.cc:398] Leaking SignalData structure 0x7b080011db80 after lost signal to thread 21718
I20240506 15:15:14.430734 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:14.906999 20952 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.010s sys 0.012s
W20240506 15:15:15.165484 20952 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.010s sys 0.013s
I20240506 15:15:14.404748 23534 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.167400 23534 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.402151 23531 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.169200 23531 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.987670 23555 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.170020 23555 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.154371 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:15.171730 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:14.987998 23560 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.172564 23560 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.174805 23560 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.988521 23559 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.175554 23559 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.178350 23559 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.988912 23558 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.179092 23558 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.181521 23558 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:14.999807 20846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.002s sys 0.025s
W20240506 15:15:15.182293 20846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.003s sys 0.025s
W20240506 15:15:15.027984 20845 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.006s sys 0.007s
W20240506 15:15:15.186991 20845 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.006s sys 0.007s
I20240506 15:15:15.190439 23331 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.191406 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.039779 23562 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.192003 20883 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:15.192121 23562 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.192947 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.194028 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.194571 23562 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.195144 23562 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.046942 23564 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.196743 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.196543 23564 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.033954 23561 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.198122 23561 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.048136 23563 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.080065 23550 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.096421 23527 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:15.096336 23517 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.097831 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.098891 23519 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:15.101400 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 873530us
I20240506 15:15:15.103101 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.106427 23549 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.107501 23572 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.108601 23573 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.116798 23571 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.116426 23574 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.064658 23505 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.118631 23569 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.119333 23567 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.126508 23575 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:14.407248 23525 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:14.401883 23530 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.143152 23576 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:15.144950 20922 consensus_peers.cc:485] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75831e0251ec4bec9c8f3ca6c4a047f0. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:15:15.145349 23568 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.199430 23564 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.240693 23568 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.240854 23572 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.217367 23511 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.242208 23563 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.243302 23568 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.190996 23331 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.119854 23570 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.151665 23145 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.244067 23569 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.244326 23145 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:14.407557 23533 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.245190 23550 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.246433 23331 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:15.246487 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 101699us
I20240506 15:15:15.247848 23550 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.248127 23569 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.244148 23570 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.248562 23519 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.243469 23571 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.249416 23574 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.250471 23570 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.246361 23533 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:14.407644 23520 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.251503 23574 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.252133 23575 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.218930 23473 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.253335 23473 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.254880 23575 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.191098 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.256143 20874 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:15.256459 23527 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.240693 23549 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.258574 23576 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.204329 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.251062 23571 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.248235 23573 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.241226 23567 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.242540 23511 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.195647 23562 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.258725 23517 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.269129 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.269384 23549 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.273609 23561 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.275301 23530 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.164049 23556 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.282032 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.296576 20877 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:15.294030 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:15.292940 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 45040.4us
I20240506 15:15:15.297727 20878 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:15.298976 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.302266 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:14.403532 23394 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.265828 23555 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.310904 23576 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.311362 23527 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.243343 23572 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.316450 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.246352 23145 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.324996 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.322808 23550 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.312503 23517 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.327332 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.327816 23473 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.324759 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.341503 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.330404 23530 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:15.344980 23563 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.331429 23511 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.345968 23549 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.346748 23570 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.346572 23549 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.336330 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.337430 23573 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:14.416800 23344 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.348114 23573 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.348404 23576 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.327325 23520 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.348639 23573 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.348978 23576 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.323137 23569 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.350241 23569 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.350973 23573 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.351212 23576 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.351521 23573 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.338217 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.333947 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.352157 23569 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.343420 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.352792 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.342605 23447 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.353438 23447 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.353847 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.354571 23394 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.354980 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.355590 23447 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.320212 23571 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.318043 23568 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.344327 23550 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.347492 23577 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.347241 23570 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.315512 23575 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.342403 23574 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.352128 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.335754 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.356312 23571 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.373582 23571 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.356537 23568 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.376235 23568 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.359062 23550 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.360000 20882 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:15.359344 23577 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.360514 23575 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.379803 23577 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.360560 23570 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.356187 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.360743 23574 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.382723 23574 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.361562 23567 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.379981 23575 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.367247 23519 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.362843 23573 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.369884 23549 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.366294 23562 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.387943 23573 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.389091 23549 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.389365 23580 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.389624 23549 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.389883 23580 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.391831 23580 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.392361 23580 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.391602 23549 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.392843 23580 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.393080 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.393479 23575 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.393599 23549 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.394068 23575 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.394832 23580 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.396508 23575 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.397238 23567 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.397697 23567 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.399783 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.400944 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.400489 23549 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.400388 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.402594 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.403048 20949 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:15.403312 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.403507 23549 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.404049 23549 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:15.404073 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.404680 23549 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.405148 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.405397 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.406008 20955 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:15.406423 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.406577 23549 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.407032 20950 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:15.407090 23549 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:15.407760 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.407713 23549 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:15.400086 23567 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.408492 20958 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:15.409464 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.410475 23549 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:15.411620 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.413599 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.414042 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.414515 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:15.421383 21714 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6b40 after lost signal to thread 19729
W20240506 15:15:15.422289 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080011fce0 after lost signal to thread 20814
W20240506 15:15:15.422605 21714 debug-util.cc:398] Leaking SignalData structure 0x7b08002b3800 after lost signal to thread 20828
W20240506 15:15:15.423471 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080028f180 after lost signal to thread 20907
W20240506 15:15:15.423770 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080036e980 after lost signal to thread 20923
W20240506 15:15:15.424549 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080031c9a0 after lost signal to thread 20982
W20240506 15:15:15.425413 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080039eb00 after lost signal to thread 21056
W20240506 15:15:15.426292 21714 debug-util.cc:398] Leaking SignalData structure 0x7b080011a0a0 after lost signal to thread 21718
I20240506 15:15:15.481737 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.484647 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.486393 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.487414 20874 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:15.488782 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.490082 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.491456 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.493090 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.493163 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.494912 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.495085 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.495973 20877 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:15.496362 20875 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:15.497243 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.497617 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.498934 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.360358 20876 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:15.499042 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.501181 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.501178 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.503134 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.504233 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.504842 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.354974 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.506287 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.507700 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.507637 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.503072 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.513607 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.518036 20879 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:15.519824 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.522042 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.522306 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.523489 20883 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:15.537923 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.540854 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.542372 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.543244 20955 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:15.544996 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.546588 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.547923 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.546697 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.540129 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.557572 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.559581 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.560747 20956 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:15.560834 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.561762 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.562703 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.568987 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.578058 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:15.564610 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.585440 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.585783 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.587347 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.587394 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.588675 20950 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:15.596575 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.599279 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.599687 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.600667 20956 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:15.601787 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.602583 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.603881 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.604629 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.606791 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.609155 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.597549 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.613305 20958 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:15.615403 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.616961 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.619521 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.619760 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.618377 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.622339 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.622331 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.626550 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.625144 20954 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:15.627837 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.629328 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.629696 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.631194 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.633679 20957 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:15.634301 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.634790 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.630941 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.632169 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.632670 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.632943 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.637521 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.639151 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.641501 20952 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 3.
I20240506 15:15:15.640864 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.642781 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:15.652842 20950 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:15.654315 21032 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 15:15:15.663534 20953 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:15.673424 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.677953 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.677572 21029 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:15.694077 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.697014 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.697345 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.698873 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.699277 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.701200 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.701620 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.702684 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.702852 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.704187 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.705009 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:15.716285 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.723997 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.724990 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:15.725440 21027 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:15:15.726236 23519 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:15.727718 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:15.722483 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.728420 23519 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:15.728876 21029 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:15.729169 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:15.730090 23519 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:15.730458 23519 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.730878 23519 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:15.730206 21654 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:15.735391 23471 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:15.744834 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.747514 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.750567 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.752893 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.754587 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.757187 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.757476 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.760185 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.761621 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.762790 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.763644 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:15.766013 23580 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:15.766464 21032 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:15:15.769405 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:15.773581 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.775055 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.774653 23580 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:15.783591 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.783939 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.785890 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.786756 21029 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:15.788309 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.788491 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.789781 21025 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:15:15.790318 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.793179 21029 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:15.799546 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.789832 23519 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.800848 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.803472 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.804018 21028 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:15.792155 20846 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:15.805835 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.809095 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.810676 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.809056 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.806205 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.813076 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.814083 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.817483 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.818529 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.819108 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.818743 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.820885 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.823239 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.824283 23527 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:15.825081 23527 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.825979 23527 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:15.830191 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.832383 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.833034 23567 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:15.833833 23519 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.845434 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.845463 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:15.851159 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.853857 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.854645 21025 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:15.851838 23567 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:15.857676 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.858904 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.858546 23567 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.866654 21031 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:15.867787 23567 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:15.887446 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:15.888008 20996 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.130:40545 timed out after 1.574s (SENT) [suppressed 25 similar messages]
I20240506 15:15:15.890107 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.890264 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.898164 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.899742 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.899425 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.905009 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.893868 23527 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:15.920348 23527 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:15.937413 20846 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:15.938025 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.939512 23519 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:15.940001 23519 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:15.940176 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:15.937682 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.959674 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.983597 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:15.958371 20998 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:15.983204 23519 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:15.997682 23508 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:15.998003 23543 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240506 15:15:15.998683 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 106809us
I20240506 15:15:15.998585 23543 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.003182 23543 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
W20240506 15:15:15.996430 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 115631us
I20240506 15:15:15.998531 23508 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.005242 23508 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
W20240506 15:15:16.016534 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 76456.7us
I20240506 15:15:16.018697 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.020428 23541 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:16.020895 23541 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:16.021273 23541 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.022572 23543 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.019088 23567 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.026244 23543 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.032367 23567 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.053061 23508 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.056471 23508 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.008937 23519 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.073683 23527 raft_consensus.cc:979] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:16.074431 23527 raft_consensus.cc:1067] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:16.079643 23527 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.080502 23527 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.081806 23543 raft_consensus.cc:979] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:16.082895 23527 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.082985 23543 raft_consensus.cc:1067] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:16.084985 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.085201 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.086643 20921 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.087126 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.087930 23567 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:15:16.089031 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 88392us
I20240506 15:15:16.089367 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.091974 20921 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.092621 23582 ts_tablet_manager.cc:902] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.093379 23549 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.095970 21656 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.097554 23542 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.101831 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:16.102015 23549 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:16.103727 23536 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.104154 20845 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.104156 23536 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.105518 23536 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.107467 23541 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.109508 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.110711 23527 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.111375 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.104241 20921 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.111544 20845 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.109758 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.101836 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.114405 23542 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:16.114480 23519 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.114634 23587 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.116324 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.116385 23586 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:16.113174 20921 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.117853 23586 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.118357 23586 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.119110 20845 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.120050 23587 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.121119 20996 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.122344 23549 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.122653 20845 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.122876 23549 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.123548 23587 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.124282 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.124132 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.123530 23549 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.125974 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.126039 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.126837 23541 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.127374 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.128916 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.127926 23589 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:16.135327 23589 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.135761 23589 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.128598 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:16.130905 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.138172 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.131376 23587 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.140960 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.129331 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.127919 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.133237 23582 tablet_copy_client.cc:316] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.130:40545
I20240506 15:15:16.144412 20845 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.145006 23567 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.157616 23567 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.158102 23567 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.158382 23541 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.145918 20921 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.146664 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.147370 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:16.162341 23577 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.162356 20921 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.164219 23579 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:16.164554 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.164688 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.165338 20953 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.158427 20845 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.148674 23585 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:16.149019 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:16.166594 23585 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.167069 23585 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.167732 23579 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.149791 20998 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:16.168975 20921 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.150321 23587 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:16.153314 20957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35816
I20240506 15:15:16.154287 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:16.156802 20996 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.147540 23586 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.116098 23536 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.183409 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.174836 23543 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:16.173537 20957 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:16.185837 20952 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:16.187131 20957 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.196000 23541 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.164027 23589 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.202971 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.175607 23577 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.206272 23584 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.206784 23584 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.207151 23584 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.207954 23579 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:16.186981 23543 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.217657 23543 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.218104 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.220458 23579 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.221402 23586 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.221940 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.223235 23589 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.225032 20953 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:16.225792 20968 tablet_copy_service.cc:140] P ecededcd854842d4867fda9fa63a4b66: Received BeginTabletCopySession request for tablet 75831e0251ec4bec9c8f3ca6c4a047f0 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:44378)
I20240506 15:15:16.225994 23586 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.226493 20968 tablet_copy_service.cc:161] P ecededcd854842d4867fda9fa63a4b66: Beginning new tablet copy session on tablet 75831e0251ec4bec9c8f3ca6c4a047f0 from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:44378: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-75831e0251ec4bec9c8f3ca6c4a047f0
I20240506 15:15:16.227281 20921 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.219905 23585 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:16.228509 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 58398.4us
I20240506 15:15:16.234297 23567 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.236497 23577 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.237121 23536 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.216125 20957 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.248095 23577 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.255664 23577 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.257589 23585 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.252182 20957 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.259212 23549 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.260452 20958 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.265916 20998 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:16.272994 23536 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.273622 23536 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.274155 23536 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.274394 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:16.275610 20998 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:16.277523 23592 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:16.277959 23592 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:16.278092 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 50915.9us
I20240506 15:15:16.278398 23592 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:16.278977 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.280300 23573 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.281644 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.282488 23573 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.284817 23536 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.287238 23592 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.298974 20968 tablet_copy_source_session.cc:205] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:15:16.288465 23585 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.304778 23585 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.276175 23549 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.305308 23585 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.290588 23543 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:16.310123 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 72416.2us
I20240506 15:15:16.311486 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.312431 23586 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:16.294652 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.292466 23567 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.319027 23583 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.321730 20998 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.283898 20921 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.299695 23592 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.323722 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.287628 23584 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.325122 20921 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.326144 23595 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.326948 23586 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:16.327572 23586 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.327957 23586 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
W20240506 15:15:16.328080 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 147840us
W20240506 15:15:16.328888 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 172147us
I20240506 15:15:16.327008 23595 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.316978 23585 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.330365 23577 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.331988 23585 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.316236 23543 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.318133 20957 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:16.331784 23541 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:16.326498 23584 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.330356 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.334549 23593 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:16.335947 23593 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.336890 23592 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.338016 23543 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.339737 20996 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.340727 23592 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:16.326462 20921 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.341447 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.342093 23595 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:16.342656 20921 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.343207 23595 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.343339 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.343833 23595 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.299917 23536 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:16.341470 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 183119us
I20240506 15:15:16.342629 23543 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:16.344807 23543 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.344923 20845 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.345496 23543 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.332535 23577 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.348703 23541 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.349357 23541 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.350112 20998 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.350934 23567 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:16.351429 23541 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.351775 23501 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:16.352146 20845 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.354017 23501 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.354249 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.351326 23567 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.355041 23567 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:16.356433 23583 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.355535 23501 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.357024 23501 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.357489 23501 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.357723 20996 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.358515 23585 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.353631 23593 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.355625 20845 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.361073 23593 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.361214 20996 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.362160 23587 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.363374 20845 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.361817 23586 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.372220 23587 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:16.372859 23587 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.373231 23587 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:16.375973 23582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75831e0251ec4bec9c8f3ca6c4a047f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:15:16.376359 23586 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.377826 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.378520 20880 raft_consensus.cc:3032] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:16.378808 20880 raft_consensus.cc:729] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.379514 20880 consensus_queue.cc:260] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.380244 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.380772 20880 raft_consensus.cc:3037] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.382494 23583 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.383800 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.394919 23596 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.384425 20996 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.396021 23583 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.386322 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.386435 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.396704 23594 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.387176 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.387879 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.388219 23549 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Committing config change with OpId 5.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
W20240506 15:15:16.399145 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 58990.5us
I20240506 15:15:16.395318 20845 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.397902 23596 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240506 15:15:16.381434 23543 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.404958 23541 raft_consensus.cc:2726] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.405442 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.381434 23587 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.408133 23587 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.408022 20845 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.408823 23501 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
W20240506 15:15:16.393983 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 51458.3us
I20240506 15:15:16.396459 23583 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.406394 21029 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:16.420414 23583 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.409480 23543 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.422969 23597 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240506 15:15:16.424444 23597 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:16.424865 23597 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:16.425961 23597 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.427407 23549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44376 (request call id 4) took 4357 ms. Trace:
I20240506 15:15:16.427651 23597 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.427873 23549 rpcz_store.cc:276] 0506 15:15:12.069658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:12.076863 (+ 7205us) service_pool.cc:225] Handling call
0506 15:15:12.082203 (+ 5340us) log.cc:842] Serialized 506 byte log entry
0506 15:15:16.427284 (+4345081us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1833848,"raft.queue_time_us":2519943,"raft.run_cpu_time_us":1372,"raft.run_wall_time_us":1370,"thread_start_us":1249080,"threads_started":2,"wal-append.queue_time_us":291930,"wal-append.run_cpu_time_us":1790,"wal-append.run_wall_time_us":1001818}
I20240506 15:15:16.430230 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.431597 20921 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.432387 23586 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:16.432793 23586 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.433184 23586 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.433589 23577 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.393517 20880 raft_consensus.cc:2454] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:16.411929 23594 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.436564 23594 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.407941 23567 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.437022 23594 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.414710 23595 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.416446 23587 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.409799 20845 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.443217 23587 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.446545 20845 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.447819 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.448606 23587 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.448923 20921 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.449741 23580 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.450273 23587 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.450903 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.451187 20845 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.452920 23596 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.411710 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.453027 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.454659 23596 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.455325 23587 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.455487 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:16.457149 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:16.458640 20845 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.459646 23586 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.460647 23587 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.462466 23586 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.463826 20845 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.429109 23583 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.463069 23567 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.460700 20846 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.469494 23583 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.470459 23587 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:16.473012 23587 raft_consensus.cc:2781] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:16.473626 23587 raft_consensus.cc:686] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.473532 23562 consensus_queue.cc:1046] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:16.416095 20998 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.474944 23587 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.476049 23588 raft_consensus.cc:3032] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:16.476505 23588 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.477305 23594 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.477456 23588 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.472106 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:16.479403 23594 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.479789 20845 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.482841 23583 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.484416 23527 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.485597 20921 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.486930 23577 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.453008 23501 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:16.476271 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60247.8us
I20240506 15:15:16.487568 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.488276 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.488822 20949 raft_consensus.cc:1226] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.489037 20955 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 10. Current term is 11. Ops: []
I20240506 15:15:16.489122 20953 raft_consensus.cc:1226] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.489795 20998 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.490031 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.490664 23585 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:16.484483 23580 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.491379 20958 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.492610 23585 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:16.493046 23585 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.493602 23583 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.495733 20998 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.496620 23588 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.498240 20924 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.499220 21689 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:16.500490 23567 raft_consensus.cc:2781] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:16.501292 23567 raft_consensus.cc:686] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.502251 23567 consensus_queue.cc:237] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.503794 23508 consensus_queue.cc:1046] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:16.504349 23527 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:15:16.505013 23508 consensus_queue.cc:1046] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:16.505439 23527 consensus_queue.cc:1046] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:16.505869 23519 raft_consensus.cc:3032] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:16.506175 23508 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Stepping down as leader of term 10
I20240506 15:15:16.506323 23519 raft_consensus.cc:729] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.506600 23508 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.506712 23596 raft_consensus.cc:3032] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:16.507220 23596 raft_consensus.cc:729] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.507328 23519 consensus_queue.cc:260] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.507953 23541 raft_consensus.cc:3032] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:16.508528 23541 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.508147 23596 consensus_queue.cc:260] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.509006 23519 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.509774 23596 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.509405 23541 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.513473 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.513960 20882 raft_consensus.cc:1226] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.515947 23541 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.516068 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.486816 20921 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.471766 23595 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:16.481086 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.521189 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.475709 21029 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:16.522109 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.522574 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:16.524816 23501 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.525733 23527 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.530880 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.531137 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.531967 21025 raft_consensus.cc:2379] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:16.532596 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.535254 21027 raft_consensus.cc:1226] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:16.536087 20881 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:16.536617 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.483994 20779 catalog_manager.cc:5484] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:16.536971 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.537348 21032 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 10: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:16.507568 23508 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.543118 23582 tablet_copy_client.cc:791] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:15:16.544158 23582 tablet_copy_client.cc:655] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
W20240506 15:15:16.544644 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57892.9us
I20240506 15:15:16.560693 20921 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.545105 23508 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Advancing to term 11
W20240506 15:15:16.545461 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49804.8us
I20240506 15:15:16.561945 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.563153 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.564116 23579 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.564783 23579 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.565338 23579 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.545210 23580 consensus_queue.cc:1046] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:16.566619 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.539712 23585 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.567981 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.568773 23585 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:16.546826 23543 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.569214 23595 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:16.566996 23580 raft_consensus.cc:3032] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:16.570120 23580 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.571363 23595 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.571611 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.551700 20957 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:16.571027 23580 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:16.573022 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.556736 23586 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.573617 23580 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.559055 23594 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:16.558916 23597 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240506 15:15:16.555900 20876 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:16.589217 20880 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:16.583084 20845 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.576397 23527 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.582546 23593 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.585610 23594 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:16.597457 23587 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240506 15:15:16.611467 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.611981 21032 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.612697 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.610126 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:16.615346 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.584666 23596 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20240506 15:15:16.616235 20957 raft_consensus.cc:1226] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:16.636487 23596 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:16.636955 23596 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.637814 23596 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.649788 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.652807 23594 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.658684 20879 raft_consensus.cc:2454] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:16.687057 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:16.687412 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:16.690148 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 115637us
I20240506 15:15:16.699884 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.701661 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.703256 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.700585 20996 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.709036 20924 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:16.704531 23542 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:16.709929 23542 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:16.710733 23542 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.712704 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.714001 20780 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 6 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:16.716420 23573 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:16.718017 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:16.719580 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 114802us
I20240506 15:15:16.721105 20879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:16.712139 23542 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:16.721617 20879 raft_consensus.cc:465] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Not starting forced leader election -- already a leader
I20240506 15:15:16.725401 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.725500 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.717211 23573 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.732854 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.726972 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.732987 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.727871 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.722815 23579 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election won for term 11
I20240506 15:15:16.727324 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.734833 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.736244 23579 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.722898 23595 raft_consensus.cc:2781] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:16.735975 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.737448 20878 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 7: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:16.750082 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.752866 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.756170 23573 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.758512 23596 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:16.737287 23579 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:16.766122 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 73484.6us
I20240506 15:15:16.768302 20949 raft_consensus.cc:2396] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20240506 15:15:16.780373 20874 raft_consensus.cc:1226] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.787225 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.782562 23603 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.789862 21027 raft_consensus.cc:1226] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:16.791281 21031 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.792500 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:16.792798 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.793956 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:16.774935 23595 raft_consensus.cc:686] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.801424 23595 consensus_queue.cc:237] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.805507 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.808638 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.809044 21030 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:16.808815 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.809602 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.810039 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.810282 21029 raft_consensus.cc:2379] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:16.811168 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.811300 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.812384 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.814522 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:16.814774 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.815114 21028 raft_consensus.cc:1226] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:16.815582 23584 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:16.815984 23584 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.816421 23584 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.817163 21032 raft_consensus.cc:1226] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240506 15:15:16.807651 20778 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 7 to 8, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:16.825706 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.828917 20996 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.830411 20996 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.831291 23585 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.831770 20996 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.831944 23585 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.835018 23584 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.848865 21027 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.844985 23594 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:16.852080 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.861797 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.862286 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.863526 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:16.864171 21029 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:16.864552 21026 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:16.865832 21032 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.870923 23584 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.881697 20996 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.884155 20996 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.882771 20879 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:16.887231 23603 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:15:16.887656 23603 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.888007 23603 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:16.893677 23585 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:16.894419 23585 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:16.895422 23573 consensus_queue.cc:1046] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:16.897001 23611 raft_consensus.cc:3032] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:16.897426 23611 raft_consensus.cc:729] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.897778 23573 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:16.898242 23611 consensus_queue.cc:260] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.899773 23611 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:16.899648 23567 raft_consensus.cc:3032] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:16.900485 23567 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.901746 23567 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.903702 23567 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:16.904407 23603 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.906567 23603 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.899891 20996 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.908417 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:16.909628 20996 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:16.910951 23585 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.911280 23536 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:16.911650 23585 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:16.911705 23536 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:16.912148 23585 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:16.912721 23585 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.912206 23536 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:16.916729 20996 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:16.921264 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 200896us
I20240506 15:15:16.928865 23583 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.928988 20921 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.930683 23595 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.931716 20921 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.933187 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.934610 20958 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:16.934979 23618 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:16.935878 23618 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:16.938407 20921 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.940049 23618 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.940758 23618 consensus_queue.cc:1046] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:16.942106 23577 raft_consensus.cc:3032] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:16.942488 23577 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:16.943354 23577 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.944182 23582 tablet_copy_client.cc:531] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:15:16.944954 23577 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.951496 20879 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:48226
I20240506 15:15:16.952353 20879 raft_consensus.cc:595] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.952716 20879 raft_consensus.cc:603] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:16.955844 23501 consensus_queue.cc:1046] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:16.957000 23501 raft_consensus.cc:3032] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:16.957356 23501 raft_consensus.cc:729] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.958206 23501 consensus_queue.cc:260] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.959981 23501 raft_consensus.cc:3037] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:16.971385 20921 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.973415 23536 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:16.961747 23584 consensus_queue.cc:1046] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:16.963928 20876 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:16.974463 23618 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:16.974260 23596 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:16.979744 23584 raft_consensus.cc:3032] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:16.976887 20874 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:16.980233 23527 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:16.980336 23584 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:16.976887 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:16.980872 23527 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:16.981488 23584 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.981905 23527 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.983064 23584 raft_consensus.cc:3037] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:16.983577 23527 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:16.975409 20921 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:16.985782 23585 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:16.987303 23536 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:16.989353 20998 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.992960 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:16.961298 23582 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:15:17.009083 20874 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:17.004856 23613 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:17.007226 21026 raft_consensus.cc:1226] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:17.011435 21031 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:17.002874 20996 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.000514 23603 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:17.008160 23585 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:17.013360 20876 raft_consensus.cc:1226] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:17.024487 23584 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:17.025920 23579 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:17.027323 20957 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:17.030027 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:17.031041 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:17.032114 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:17.019131 23611 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:17.034464 23595 consensus_queue.cc:1046] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:17.044872 23501 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240506 15:15:17.047461 20996 consensus_peers.cc:485] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6abe388354e4b7e8f0656dda82078bc. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:15:17.045460 23595 raft_consensus.cc:3032] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:17.039175 20779 catalog_manager.cc:5484] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 6 to 7, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.048465 23595 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:17.034020 20921 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:17.044185 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 51326.3us
I20240506 15:15:17.051324 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.051540 21031 raft_consensus.cc:1226] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:17.051993 23618 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:17.051328 20921 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:17.049757 20779 catalog_manager.cc:5484] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 10 to 11, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 11 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.053253 20998 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.057714 23604 consensus_queue.cc:1046] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:17.058084 23562 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:17.058904 23604 raft_consensus.cc:3032] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:17.059283 23604 raft_consensus.cc:729] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:17.059511 23567 consensus_queue.cc:1046] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:17.056073 23613 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:17.054339 23579 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:17.060916 23618 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:17.049731 23595 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.062196 23618 consensus_queue.cc:1046] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:17.060286 23604 consensus_queue.cc:260] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:17.063588 23595 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.063767 23567 raft_consensus.cc:3032] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:17.064587 23573 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:17.065335 23573 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:17.065807 23580 raft_consensus.cc:3032] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:17.066220 23580 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:17.066488 23573 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.067032 23580 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.068913 23580 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:17.069638 23603 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:17.070210 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.071031 23573 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:17.071564 23567 raft_consensus.cc:729] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:17.074003 20998 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.077381 20779 catalog_manager.cc:5484] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 6 to 7, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.081303 23536 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240506 15:15:17.082620 23567 consensus_queue.cc:260] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.083189 21684 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240506 15:15:17.085610 20998 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.088490 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.064347 23604 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:17.089903 23549 consensus_queue.cc:1046] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:17.090245 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.091153 20958 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:17.117272 23549 raft_consensus.cc:3032] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:17.091183 20953 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:17.089323 23585 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:17.093500 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.092880 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:17.120204 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.121379 23585 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:15:17.094108 20779 catalog_manager.cc:5484] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 4 to 5, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 5 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.105577 21031 raft_consensus.cc:1226] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:17.108589 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.109165 23536 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240506 15:15:17.115231 23613 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:17.117772 23549 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:17.123876 23549 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.125315 23549 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.128998 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.131927 23562 consensus_queue.cc:1046] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:17.132200 23613 consensus_queue.cc:1046] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:17.133189 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.133914 23567 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:17.130273 23584 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:17.133740 23536 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:17.140173 23613 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:17.146068 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.144091 20998 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.144405 23613 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:17.160154 23613 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.162643 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.164227 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.165021 23603 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:17.182286 23527 raft_consensus.cc:3032] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:17.182751 23527 raft_consensus.cc:729] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:17.179638 23603 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:17.169332 23592 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:17.178532 23589 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:17.170708 23584 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:17.178383 23583 consensus_queue.cc:1046] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240506 15:15:17.169543 23613 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.183629 23527 consensus_queue.cc:260] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.212105 21026 raft_consensus.cc:1226] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:17.216554 23543 raft_consensus.cc:3032] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:17.234896 20949 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:17.224593 23543 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:17.275616 20882 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240506 15:15:17.288379 23605 consensus_queue.cc:1046] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20240506 15:15:17.274168 23527 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:17.358505 23628 ts_tablet_manager.cc:902] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:17.323777 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 44880.3us
I20240506 15:15:17.364077 21032 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:17.365046 20949 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240506 15:15:17.366582 23613 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240506 15:15:17.338024 23579 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
W20240506 15:15:17.409888 23625 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240506 15:15:17.358525 23543 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.417587 23543 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:17.384918 23597 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20240506 15:15:17.426081 20874 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:17.432317 23567 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240506 15:15:17.438382 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 274194us
I20240506 15:15:17.442386 20876 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240506 15:15:17.459517 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 74127.7us
I20240506 15:15:17.480484 23632 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240506 15:15:17.481767 23632 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:17.482156 23632 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:17.483031 23632 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.458817 23549 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240506 15:15:17.477375 23585 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.489331 23469 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:17.489804 23469 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:17.490672 23469 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:17.493847 23469 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:17.498139 20957 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.516959 23632 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:15:17.525022 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 48980.6us
I20240506 15:15:17.525995 23584 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:17.527122 23584 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:17.527513 23584 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:17.528721 23584 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.530874 23584 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:17.539261 23585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:49614 (request call id 8) took 1066 ms. Trace:
I20240506 15:15:17.526154 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.546192 23613 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:17.541399 23585 rpcz_store.cc:276] 0506 15:15:16.472989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:16.473145 (+ 156us) service_pool.cc:225] Handling call
0506 15:15:16.526152 (+ 53007us) log.cc:842] Serialized 506 byte log entry
0506 15:15:16.695063 (+168911us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b1800191f20. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b33e2ea 00007f6314a2dfe5 00007f630b3c23d4 00007f630b3c1cc5 00007f630e269570 00007f630e2cf363 00007f630e2f5a32 00007f6313232cd9 00007f630c91a714
0506 15:15:16.717141 (+ 22078us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b5400075ba0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e280c9f 00007f630e269795 00007f630e26e9b1 00007f630e26e939 00007f630e26e8c9 00007f630e26e891 00007f630e26da5d 00007f6314639da4 00007f6314639bd9 00007f630b3c5236
0506 15:15:16.746873 (+ 29732us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b1800191f20. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b33e2ea 00007f6312199898 00007f630b3c5336 00007f630b3c8691 00007f630b3c8649 00007f630b3c85d9 00007f630b3c85a1 00007f630b3c789d 00007f6314639da4
0506 15:15:17.539126 (+792253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155209,"raft.queue_time_us":77402,"raft.run_cpu_time_us":3112,"raft.run_wall_time_us":65635,"spinlock_wait_cycles":31580672,"thread_start_us":96435,"threads_started":1,"wal-append.queue_time_us":182610}
I20240506 15:15:17.560523 21029 consensus_queue.cc:237] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:17.585230 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:17.580938 23628 tablet_copy_client.cc:316] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:15:17.590433 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet f6abe388354e4b7e8f0656dda82078bc from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:15:17.591018 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet f6abe388354e4b7e8f0656dda82078bc from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-f6abe388354e4b7e8f0656dda82078bc
I20240506 15:15:17.578109 23583 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240506 15:15:17.611800 23542 raft_consensus.cc:3032] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:17.612222 23542 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:17.613168 23542 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:17.614501 23542 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:17.627619 21042 tablet_copy_source_session.cc:205] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:15:17.630864 20780 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e reported cstate change: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 5 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.635716 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6abe388354e4b7e8f0656dda82078bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:15:17.638303 21032 raft_consensus.cc:1261] T 4c9c7ee4280049d091aea89be873b5c8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:17.652473 21026 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:17.654000 21031 raft_consensus.cc:3037] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:17.654093 21027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:17.666424 21027 raft_consensus.cc:595] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:17.666800 21027 raft_consensus.cc:603] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:17.654021 21028 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:17.660128 23501 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:17.669029 20874 raft_consensus.cc:2454] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:17.676564 23567 consensus_queue.cc:1035] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:17.687628 21026 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:17.688534 20875 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
W20240506 15:15:17.703469 20996 consensus_peers.cc:485] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f4d07ea30646d0805b63bb641358be. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:15:17.717787 23579 consensus_queue.cc:1035] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:17.719501 21028 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240506 15:15:17.720285 20957 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:15:17.669570 23501 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.723116 20998 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.725296 20876 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240506 15:15:17.726176 23567 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:15:17.727424 23584 raft_consensus.cc:2781] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:17.728008 23584 raft_consensus.cc:686] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:17.729341 23501 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.728956 23584 consensus_queue.cc:237] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.731981 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:17.732293 21031 raft_consensus.cc:1261] T 629a891c5ba546aeb0e0027a02f1b955 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240506 15:15:17.732839 20957 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:17.734380 23632 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:17.734534 20846 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.735920 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:17.737191 23579 consensus_queue.cc:1035] T 629a891c5ba546aeb0e0027a02f1b955 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:17.739467 23582 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 15:15:17.740559 23582 tablet_bootstrap.cc:492] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:15:17.740341 23501 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:17.741426 23582 ts_tablet_manager.cc:1367] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.780s user 0.016s sys 0.038s
I20240506 15:15:17.741668 23501 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:17.742399 23501 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.743072 23632 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:17.744740 23582 raft_consensus.cc:348] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.751188 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:17.752632 21028 raft_consensus.cc:595] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Received request to transfer leadership to TS ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:17.757452 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240506 15:15:17.758031 23585 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.759280 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:17.761188 20950 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.760069 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:17.780086 20874 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:17.778301 23501 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.786257 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:17.790344 23501 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.793745 23541 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:17.793996 21032 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:17.794248 23541 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.795689 23527 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:17.796207 23541 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.795406 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.797271 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:17.802685 20957 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:17.799651 20780 catalog_manager.cc:5484] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 4 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.804059 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:17.804745 20955 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:17.807533 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:17.798254 20950 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.809818 23501 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:17.810251 23501 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:17.810592 23501 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
W20240506 15:15:17.809842 20846 consensus_peers.cc:485] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a -> Peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421): Couldn't send request to peer 7c8800c6dcaf4b6e8dd58c4149f73c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d096daea6fa4799a608705b3dc732db. This is attempt 1: this message will repeat every 5th retry.
I20240506 15:15:17.818048 20876 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.825867 23582 raft_consensus.cc:729] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:15:17.826689 23582 consensus_queue.cc:260] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:17.833681 23527 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:17.839088 23582 ts_tablet_manager.cc:1398] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.097s user 0.004s sys 0.004s
I20240506 15:15:17.841290 20950 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:17.845144 20968 tablet_copy_service.cc:342] P ecededcd854842d4867fda9fa63a4b66: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-75831e0251ec4bec9c8f3ca6c4a047f0 received from {username='slave'} at 127.0.0.1:44378
I20240506 15:15:17.848583 20968 tablet_copy_service.cc:433] P ecededcd854842d4867fda9fa63a4b66: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-75831e0251ec4bec9c8f3ca6c4a047f0 on tablet 75831e0251ec4bec9c8f3ca6c4a047f0 with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:17.852437 20846 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:17.853669 23596 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:17.854173 23596 raft_consensus.cc:686] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:17.853914 23541 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.855032 23596 consensus_queue.cc:237] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.859239 23628 tablet_copy_client.cc:791] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:15:17.860154 23628 tablet_copy_client.cc:655] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:15:17.866596 20950 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.869974 21028 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.888346 23628 tablet_copy_client.cc:531] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:15:17.889881 23527 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:17.886487 20779 catalog_manager.cc:5484] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:17.903203 23541 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:17.903888 23541 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.906744 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:17.907456 20957 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:17.908552 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.909432 23596 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:17.910163 23596 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:17.910528 23596 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.890429 23527 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.876212 20780 catalog_manager.cc:5484] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 7 to 8, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:17.920024 23541 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.921041 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:17.921684 21032 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:17.924310 23628 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:15:17.919193 23596 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.928791 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.933161 20950 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.939011 23596 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.941982 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:17.942693 20957 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:17.943847 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.944766 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:17.945976 23579 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:17.946568 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:17.946593 23579 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:17.947705 23527 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.949159 23579 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.943956 20780 catalog_manager.cc:5484] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 reported cstate change: config changed from index -1 to 6, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:17.877604 23501 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.947358 21031 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:17.951912 21028 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 6
I20240506 15:15:17.952586 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:17.953266 21032 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:17.954260 20921 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.954989 23579 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:17.955348 23579 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:17.955662 23579 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:17.959606 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:17.945559 23596 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:17.959935 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:17.960335 20876 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:17.960441 23596 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:17.960690 20957 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:17.960856 23596 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:17.966954 23654 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20240506 15:15:17.970460 23579 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:17.971417 23501 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.972386 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:17.978047 20950 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:17.979317 20846 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.972477 20957 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:17.983106 20846 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:17.984382 23541 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:17.985170 23541 raft_consensus.cc:686] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:17.986122 23541 consensus_queue.cc:237] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:17.988772 23567 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:17.989392 23567 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:17.990398 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:17.990998 20876 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:17.991392 23567 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:17.993036 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:17.993692 21684 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:17.995092 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:17.995667 21032 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:17.996187 23579 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:17.996608 20922 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:17.998181 21028 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:17.999050 23567 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:18.001577 23527 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:18.001760 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.004810 23567 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:18.005012 23527 raft_consensus.cc:686] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:18.005317 23567 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:18.005244 20875 raft_consensus.cc:2379] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 10: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:18.002110 20876 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:18.004331 21032 raft_consensus.cc:2454] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:18.007779 20924 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.008579 23577 raft_consensus.cc:2781] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:18.009081 23577 raft_consensus.cc:686] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:18.009927 23577 consensus_queue.cc:237] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.006163 23527 consensus_queue.cc:237] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.020879 23567 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.019709 23501 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.022357 23596 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.026861 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:18.045425 20875 raft_consensus.cc:2379] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 10: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:18.034067 23584 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:18.025321 23567 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:18.065632 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:18.077167 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:18.077824 20950 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:18.064533 23584 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.087813 23596 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.088183 23584 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.085065 20779 catalog_manager.cc:5484] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.089248 20950 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:18.093421 20846 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.094991 23587 raft_consensus.cc:2781] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:18.095481 23587 raft_consensus.cc:686] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:18.096545 23587 consensus_queue.cc:237] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.030364 23501 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.089393 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.090731 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.091378 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.099452 21028 raft_consensus.cc:3037] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 7
I20240506 15:15:18.100992 20955 raft_consensus.cc:2354] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
I20240506 15:15:18.106453 23501 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.111029 21028 raft_consensus.cc:2454] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:18.112893 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.113889 20955 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:18.115067 20780 catalog_manager.cc:5484] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.116075 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.121325 20996 leader_election.cc:398] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
I20240506 15:15:18.121861 20996 leader_election.cc:401] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:18.124591 23584 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.127789 23501 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:18.128422 23501 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.128849 23501 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:15:18.154647 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57391us
I20240506 15:15:18.150626 23587 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.157341 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.158476 21028 raft_consensus.cc:2379] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.161451 23587 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.182235 23567 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.189069 23587 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:18.189313 23567 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.191726 23567 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.192694 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:18.182545 21684 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:18.195111 20922 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:18.196326 23567 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:18.197031 23567 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:18.197894 23567 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.200649 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.202054 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.204839 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:18.208635 23501 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.209599 23584 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6.
I20240506 15:15:18.210929 23501 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.211571 23501 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.212623 23501 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.213768 23596 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.214275 23596 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.214824 23501 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.214973 23541 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:18.216632 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:18.216881 23596 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.217350 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.218799 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.219444 20957 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:18.221477 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.222455 23596 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:18.222852 23596 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.223227 23596 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.224417 20950 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:18.226861 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.227743 23541 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:18.228230 23541 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:18.229146 23541 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.223126 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.238466 21028 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:18.239496 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:18.240821 23656 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:18.241261 23656 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:18.241586 23656 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:18.243981 23542 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.244654 23542 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.246860 23542 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.249754 23656 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.252774 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:18.255970 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 105670us
I20240506 15:15:18.258643 21028 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:15:18.261693 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:18.262136 23613 raft_consensus.cc:979] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: : Instructing follower ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:18.262660 23584 raft_consensus.cc:1067] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Signalling peer ecededcd854842d4867fda9fa63a4b66 to start an election
I20240506 15:15:18.256795 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.262672 23542 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:18.263876 23542 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:18.264238 23542 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:18.265785 20955 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:18.270056 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:18.270954 23656 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.272150 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.272776 21028 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:18.273016 20779 catalog_manager.cc:5484] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.277556 23542 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.296068 20957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
from {username='slave'} at 127.0.0.1:35810
I20240506 15:15:18.296706 20957 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:18.297072 20957 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.299880 23542 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.299849 20780 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 9 to 10, leader changed from 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 10 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.305846 20957 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.306560 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.307196 21032 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:18.309412 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:18.310242 20875 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.315488 23582 ts_tablet_manager.cc:902] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.258762 23596 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.323194 21032 raft_consensus.cc:2454] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:15:18.324688 21653 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:18.325459 23542 raft_consensus.cc:2781] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:18.326570 23542 raft_consensus.cc:686] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:18.327589 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:18.326754 23662 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:18.328230 23542 consensus_queue.cc:237] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.284175 20779 catalog_manager.cc:5484] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.335569 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.336242 21032 raft_consensus.cc:3032] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:18.336556 21032 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:18.337275 21032 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.337759 20875 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
W20240506 15:15:18.338654 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 163586us
I20240506 15:15:18.338943 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.340447 23656 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:18.340974 23656 raft_consensus.cc:686] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:18.342061 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.341859 23656 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.342906 21031 raft_consensus.cc:2379] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.344921 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.345502 21031 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:18.347698 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:18.348940 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.349696 21027 raft_consensus.cc:2379] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.349645 23541 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:18.350322 23541 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.354588 20949 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:18.355314 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.356069 20955 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:18.357630 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.358898 23664 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:18.359622 23664 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:18.360457 23664 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.352389 23541 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.357954 20957 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.377840 23596 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.380502 23577 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.394430 23668 ts_tablet_manager.cc:902] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
I20240506 15:15:18.394320 20779 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 5, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 5 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.398084 23628 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:15:18.387622 23582 tablet_copy_client.cc:316] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.131:42913
I20240506 15:15:18.393201 21028 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:18.399847 21032 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.401466 20921 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:18.391193 23577 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.403175 23628 tablet_bootstrap.cc:492] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:15:18.399540 23664 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.408077 21042 tablet_copy_service.cc:140] P 7610624a16b44896b6aacea1928ab20e: Received BeginTabletCopySession request for tablet a9f4d07ea30646d0805b63bb641358be from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49620)
I20240506 15:15:18.401209 20778 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 6, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.414578 20875 raft_consensus.cc:1261] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:18.424300 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.425271 21042 tablet_copy_service.cc:161] P 7610624a16b44896b6aacea1928ab20e: Beginning new tablet copy session on tablet a9f4d07ea30646d0805b63bb641358be from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49620: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-a9f4d07ea30646d0805b63bb641358be
I20240506 15:15:18.423079 23579 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:18.426457 23579 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:18.427469 23577 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.427278 23579 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.428140 23577 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:18.428645 23577 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:18.419294 23656 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.435861 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.436683 21027 raft_consensus.cc:2379] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.437525 21684 raft_consensus.cc:3037] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEARNER]: Advancing to term 7
I20240506 15:15:18.440518 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.441391 21027 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:18.444175 20921 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:18.451745 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.452482 21028 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:18.452785 21027 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:18.454193 20779 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 6, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.463980 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:18.468173 21032 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:18.470029 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.470857 21029 raft_consensus.cc:2354] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:18.471266 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:18.447345 23628 ts_tablet_manager.cc:1367] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 0.523s user 0.036s sys 0.005s
I20240506 15:15:18.439271 23549 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:18.440028 23577 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:18.476985 23658 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.479359 23541 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:18.479477 23658 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.420441 23664 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.462854 23596 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.469583 23587 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.472582 23501 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.472822 23527 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:18.494314 23527 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:18.494833 23527 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:18.480051 23541 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:18.435861 23656 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.489811 23618 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:18.480504 23549 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.490200 23587 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.492424 23501 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.489094 23596 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.492893 20780 catalog_manager.cc:5484] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:18.487506 23668 tablet_copy_client.cc:316] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.67.129:44145
I20240506 15:15:18.482420 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:18.518234 23596 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.512013 20875 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:18.522403 23587 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.522645 23677 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:18.523763 20955 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240506 15:15:18.524331 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.528848 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:18.531148 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.531953 20876 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Committing config change with OpId 5.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:18.532406 20874 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:18.541311 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.508455 23541 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.510900 23618 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:18.540230 20955 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:18.537060 23675 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:18.549535 23675 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.561179 23675 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.562008 23675 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:18.562821 23549 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.563197 23675 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:18.563848 23675 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:18.565256 21042 tablet_copy_source_session.cc:205] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:15:18.548509 23618 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:18.535284 23587 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20240506 15:15:18.575481 23658 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.578191 23658 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.580893 23658 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.581480 23628 raft_consensus.cc:348] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.582463 23628 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:15:18.583442 23658 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.584738 23675 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.587388 23675 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.590867 21684 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:18.593271 23501 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.595978 23664 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.598233 23656 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.604522 23582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f4d07ea30646d0805b63bb641358be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:15:18.609588 21688 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240506 15:15:18.612181 23567 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240506 15:15:18.583148 23628 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:18.619603 23664 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:15:18.702526 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 231319us
W20240506 15:15:18.798573 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 354496us
I20240506 15:15:18.813829 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.661190 23632 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.822599 21027 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240506 15:15:18.824230 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 283018us
I20240506 15:15:18.824998 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.826216 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:18.826552 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.660894 23613 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.830454 21027 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.737947 23658 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.835089 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:18.709834 23656 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.852510 23656 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.854593 23656 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.736168 23628 ts_tablet_manager.cc:1398] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.258s user 0.007s sys 0.001s
W20240506 15:15:18.722216 23672 log.cc:924] Time spent T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240506 15:15:18.846515 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.850265 20879 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:18.811488 23661 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.856904 23661 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.755348 23587 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.858289 23587 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.858923 23661 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.684187 23585 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.860244 23585 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.861346 23587 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.718325 23675 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:18.831596 23658 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.862632 23680 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:18.862521 23675 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.863397 23680 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.864512 23658 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.713027 23676 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.161s
I20240506 15:15:18.865867 23680 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.866452 23577 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.832082 23632 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.867051 23577 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.868573 23685 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.869175 23577 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.869143 23685 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.740367 23662 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:18.874212 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:18.864969 23675 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.872422 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:18.880499 20875 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:18.648533 23541 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.874617 21026 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:18.858814 20778 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:18.646034 23664 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.897003 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.761699 23527 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.739105 23618 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.907850 21027 raft_consensus.cc:2354] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:18.862509 23585 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.915402 20892 tablet_copy_service.cc:140] P 00fe71c1675441379f5f2c7490f9e63a: Received BeginTabletCopySession request for tablet 6d096daea6fa4799a608705b3dc732db from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e ({username='slave'} at 127.0.0.1:49818)
I20240506 15:15:18.916194 20892 tablet_copy_service.cc:161] P 00fe71c1675441379f5f2c7490f9e63a: Beginning new tablet copy session on tablet 6d096daea6fa4799a608705b3dc732db from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e at {username='slave'} at 127.0.0.1:49818: session id = 7c8800c6dcaf4b6e8dd58c4149f73c4e-6d096daea6fa4799a608705b3dc732db
I20240506 15:15:18.836272 23613 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.882544 23587 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:18.918936 20874 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:18.919734 23541 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.920655 23587 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.921164 23587 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:18.921876 23618 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.882618 23662 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.924398 23662 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.924746 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.925527 21030 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.927726 23613 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:18.929106 23683 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240506 15:15:18.930681 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.931527 21029 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.932543 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.933320 21027 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:18.933665 23632 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.934243 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.934617 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.934878 21023 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:18.935528 21031 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240506 15:15:18.935808 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.936556 21030 raft_consensus.cc:2379] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 9: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.938396 23684 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:18.939034 23684 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.941169 23684 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.941511 23664 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.946107 23527 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.832712 23691 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20240506 15:15:18.935233 21029 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:18.949813 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:18.950552 21028 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:18.878312 23685 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:18.917204 23583 consensus_queue.cc:1035] T e4ed6a4c20ff49de9f94f4e07d3269ef P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240506 15:15:18.957844 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-f6abe388354e4b7e8f0656dda82078bc received from {username='slave'} at 127.0.0.1:49620
I20240506 15:15:18.958336 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-f6abe388354e4b7e8f0656dda82078bc on tablet f6abe388354e4b7e8f0656dda82078bc with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:18.964219 23587 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:18.965740 20892 tablet_copy_source_session.cc:205] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Tablet Copy: opened 0 blocks and 1 log segments
I20240506 15:15:18.973500 23587 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:18.976354 20957 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35794
I20240506 15:15:18.977191 20957 raft_consensus.cc:595] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Received request to transfer leadership
I20240506 15:15:18.929221 21032 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:18.933692 21025 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:15:18.934885 21026 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:19.023043 23567 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:19.023862 23693 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.023949 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.024881 20949 raft_consensus.cc:2354] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10.
I20240506 15:15:19.024607 23693 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.026978 23693 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.023744 23567 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.029812 23567 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.031365 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.032279 20955 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:19.011888 23582 tablet_copy_client.cc:791] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
I20240506 15:15:19.038255 20955 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240506 15:15:19.039861 20954 raft_consensus.cc:1261] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:19.046356 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.047320 20955 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:19.052815 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.053380 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.054126 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.054942 20949 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:19.059814 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.060635 20954 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:19.054008 20957 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:19.063984 21032 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240506 15:15:19.053489 20953 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:19.065762 20955 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:19.068655 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.070120 20955 raft_consensus.cc:2379] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:19.061820 23582 tablet_copy_client.cc:655] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:15:19.080541 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.083431 20956 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240506 15:15:19.081286 20951 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:19.093782 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.094429 20949 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:19.094512 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.095191 20875 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:19.092953 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.095762 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.095984 20952 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:19.096364 20951 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 7
I20240506 15:15:19.095388 20949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35794
I20240506 15:15:19.097762 20949 raft_consensus.cc:595] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Received request to transfer leadership
I20240506 15:15:19.097278 20921 leader_election.cc:398] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:19.098593 20921 leader_election.cc:401] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:19.098948 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:19.099218 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.101512 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.101799 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.102159 20875 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:19.100353 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.105636 21032 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:19.105942 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.106714 21028 raft_consensus.cc:2379] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.108526 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.109107 21032 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:19.102502 23689 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:19.110788 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.110652 23689 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.109985 21684 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Committing config change with OpId 5.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:19.111419 20882 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:19.112887 23689 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:19.103896 20958 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:19.114338 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.108526 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.115928 20876 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:19.119055 21026 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:19.120136 23656 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240506 15:15:19.123358 21688 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEARNER]: Advancing to term 6
I20240506 15:15:19.123818 23668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d096daea6fa4799a608705b3dc732db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 15:15:19.129300 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.120131 21032 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:19.131975 23642 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:19.132889 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.135515 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.136309 21026 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:19.137758 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.138386 21028 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:19.140818 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:19.141389 20956 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:19.141548 23693 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:19.142123 23693 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:19.142982 23693 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.144109 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:19.144912 23577 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:19.145308 23577 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:19.145660 23577 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:19.152546 20952 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:19.155771 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.157567 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.159303 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.160075 21026 raft_consensus.cc:2354] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:19.162119 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.156534 20874 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
I20240506 15:15:19.158550 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.163072 20882 raft_consensus.cc:2354] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:19.163794 20846 leader_election.cc:398] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10.
I20240506 15:15:19.164201 20846 leader_election.cc:401] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:19.163367 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.164952 20876 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:19.166388 20875 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240506 15:15:19.161906 20951 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:19.163324 20878 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:19.169202 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.169848 21030 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:19.171208 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.109153 23567 raft_consensus.cc:3037] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 8
I20240506 15:15:19.162799 21026 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:19.171911 21027 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:19.176219 21025 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:19.175729 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.177276 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.178470 21028 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:19.177285 20996 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.179559 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.179464 23677 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:19.182225 23670 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.182137 23677 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.182601 23698 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.180375 21024 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:19.176234 23658 raft_consensus.cc:3037] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 10
I20240506 15:15:19.162894 23656 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:19.185925 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.194684 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.195295 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.195955 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.196041 20954 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:19.197077 20996 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.200033 20996 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.200372 21028 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:19.200616 20921 leader_election.cc:398] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:19.200971 20921 leader_election.cc:401] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:19.200197 20846 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.207180 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.210884 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.211802 20958 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:19.213227 23632 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:19.213711 23632 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.214066 23632 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:19.199267 23587 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240506 15:15:19.203904 23549 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:19.217768 23690 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.218272 23690 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.218613 23690 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.220425 23697 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.220861 23697 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.221212 23697 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.188359 23698 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:19.224074 23698 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.225945 23582 tablet_copy_client.cc:531] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:15:19.205332 23642 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.229146 23642 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.231684 23642 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.233903 23632 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.208240 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.241652 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.242362 20949 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:15:19.241973 21030 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:19.243404 21654 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.234313 21688 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEARNER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20240506 15:15:19.235308 23700 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240506 15:15:19.248447 23501 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:19.182860 23670 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.236896 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.190188 23676 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:19.250718 23670 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.251022 23676 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.252065 23680 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:19.253574 23676 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:19.190189 23691 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20240506 15:15:19.189740 23661 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.254199 23661 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.254587 23661 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.209656 23541 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.255491 23541 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.255860 23541 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.198992 23675 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.244571 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.247383 23642 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:19.256630 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.256975 21032 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 4.
I20240506 15:15:19.248939 23501 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.256538 23675 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.235627 23694 raft_consensus.cc:465] T 4c9c7ee4280049d091aea89be873b5c8 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Not starting pre-election -- already a leader
I20240506 15:15:19.257406 21030 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 9.
I20240506 15:15:19.256587 23642 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.258447 23642 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:19.250741 21027 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.259198 20846 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.260411 21653 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:19.260831 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:19.267802 20950 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:15:19.268144 23699 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:19.209575 23664 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:19.271044 20845 leader_election.cc:398] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:19.283685 20845 leader_election.cc:401] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:19.265060 23582 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:15:19.282939 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.282482 23699 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.289687 23699 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:19.289433 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:19.282750 23664 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.290755 23664 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:19.238881 23577 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.293612 20955 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:19.274152 23691 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:19.263878 23501 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:19.283735 21024 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Committing config change with OpId 5.3: config changed from index -1 to 3, NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } }
I20240506 15:15:19.302453 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35794
I20240506 15:15:19.305341 20950 raft_consensus.cc:595] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Received request to transfer leadership
I20240506 15:15:19.307906 23684 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:19.314711 23684 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:19.315186 23684 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:19.318365 23656 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.304461 23691 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.314813 23679 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240506 15:15:19.304064 23632 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:19.309949 23675 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.339694 23656 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:19.337689 23587 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 8
I20240506 15:15:19.342324 23691 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:19.360735 20828 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for f6abe388354e4b7e8f0656dda82078bc already in progress
I20240506 15:15:19.367565 23668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59114 (request call id 156) took 1015 ms. Trace:
I20240506 15:15:19.351022 23686 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20240506 15:15:19.352195 23577 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.379132 23698 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.388046 23677 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:19.389786 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:15:19.391850 20828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f6abe388354e4b7e8f0656dda82078bc already in progress
I20240506 15:15:19.377839 23658 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10.
I20240506 15:15:19.408552 23697 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.416086 23698 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:19.427009 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 160589us
I20240506 15:15:19.431372 23697 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:19.437865 23686 consensus_queue.cc:1035] T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:19.441430 23686 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 15:15:19.449615 23567 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8.
I20240506 15:15:19.450804 20951 raft_consensus.cc:1261] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240506 15:15:19.451609 23686 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:19.454919 23670 consensus_queue.cc:1035] T 9178d1bf59624e7f8948c33552e74a00 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:19.456547 23686 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:19.426381 20778 catalog_manager.cc:5484] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 reported cstate change: term changed from 5 to 6, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to ecededcd854842d4867fda9fa63a4b66 (127.19.67.130). New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:19.460777 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:19.464720 20845 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:19.469377 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:19.473918 20998 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:19.475303 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 192435us
I20240506 15:15:19.482661 21027 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 10
I20240506 15:15:19.483016 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:19.474670 23596 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.474838 23686 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:19.485924 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.531278 23690 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:19.533623 23705 log.cc:924] Time spent T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20240506 15:15:19.510952 23567 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:19.536679 23699 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.486676 23583 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:19.510047 23698 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240506 15:15:19.477422 23679 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:19.483994 23661 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.480170 23697 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240506 15:15:19.499933 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 256244us
I20240506 15:15:19.549947 21029 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20240506 15:15:19.487083 23541 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.487532 23596 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.560052 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.560971 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.561301 21032 raft_consensus.cc:2379] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.566015 23690 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.560364 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.561829 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:19.535506 23684 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.597138 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.598428 20921 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.566062 23713 raft_consensus.cc:1050] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:19.605152 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.613078 23579 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:19.613659 23579 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.614053 23579 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:19.606863 23693 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:19.616858 23693 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.617254 23693 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:19.620519 23677 raft_consensus.cc:1050] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:19.621140 23690 raft_consensus.cc:1050] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:19.573557 23686 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:19.621541 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.621518 23618 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:19.622779 20921 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.623148 23618 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:19.623785 23675 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:19.624099 20921 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.624960 23676 raft_consensus.cc:2781] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:19.625075 23541 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.624800 23577 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:19.625576 23676 raft_consensus.cc:686] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [term 10 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:19.624181 23675 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:19.626479 23676 consensus_queue.cc:237] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.627238 23675 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:19.623538 23618 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:19.631817 23618 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:19.633453 23661 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:19.579819 23697 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.634182 23697 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:19.576851 23583 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.639613 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.535553 23501 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.647392 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.642612 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.648063 20949 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 10
I20240506 15:15:19.596665 23596 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.645411 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.658219 20958 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.644693 20876 raft_consensus.cc:2454] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:19.656703 20949 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
I20240506 15:15:19.604146 23658 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:19.671923 21654 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:19.674103 20876 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20240506 15:15:19.639143 23713 raft_consensus.cc:1054] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Unable to promote non-voter 7c8800c6dcaf4b6e8dd58c4149f73c4e: Service unavailable: leader transfer in progress
I20240506 15:15:19.621575 23699 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.576488 23567 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.651105 23583 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.577451 23656 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.401127 23668 rpcz_store.cc:276] 0506 15:15:18.352031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:18.352370 (+ 339us) service_pool.cc:225] Handling call
0506 15:15:18.394968 (+ 42598us) ts_tablet_manager.cc:903] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145)
0506 15:15:19.122584 (+727616us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:15:19.216209 (+ 93625us) tablet_metadata.cc:641] Metadata flushed
0506 15:15:19.217288 (+ 1079us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:15:19.366554 (+149266us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:15:19.366964 (+ 410us) tablet_replica.cc:205] Creating consensus instance
0506 15:15:19.367377 (+ 413us) ts_tablet_manager.cc:958] Replica registered.
0506 15:15:19.367470 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":14184,"tablet-copy.queue_time_us":41486,"thread_start_us":127106,"threads_started":2}
I20240506 15:15:19.678988 21686 raft_consensus.cc:2354] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10.
I20240506 15:15:19.717063 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.717962 20877 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:19.623502 23690 consensus_queue.cc:237] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.735033 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.736897 20877 raft_consensus.cc:2354] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:19.736754 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
W20240506 15:15:19.737185 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 79692.7us
I20240506 15:15:19.747694 23675 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.748955 20950 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.752899 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.753721 21031 raft_consensus.cc:2379] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.754701 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.752936 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:19.755324 20874 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:19.756204 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.756891 20881 raft_consensus.cc:2354] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:19.756981 23684 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.757406 23656 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.759157 20953 raft_consensus.cc:595] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:19.759559 20953 raft_consensus.cc:603] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240506 15:15:19.759670 23711 log.cc:924] Time spent T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20240506 15:15:19.759855 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.720489 23658 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:19.761507 23658 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.762157 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.763196 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:19.775635 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 48867.4us
I20240506 15:15:19.775959 20950 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:19.765108 21031 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:19.769789 20874 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:19.780121 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.780807 21029 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
W20240506 15:15:19.785034 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 81875us
I20240506 15:15:19.787143 23697 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.789716 23697 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:19.789846 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:19.791714 21029 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:19.794044 23684 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:19.797626 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:19.798446 20924 leader_election.cc:398] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7.
I20240506 15:15:19.798501 21028 raft_consensus.cc:2379] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 4: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.762472 23699 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:19.778124 23668 tablet_copy_client.cc:791] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 0 data blocks...
W20240506 15:15:19.801734 23642 consensus_meta.cc:287] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.517s user 0.008s sys 0.001s
I20240506 15:15:19.774477 23675 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.819219 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:19.817171 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.820004 20952 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 9: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:19.795480 23587 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:19.821998 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.792673 23697 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:19.824563 20875 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:19.825416 21026 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:19.828516 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:19.809466 23656 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.766135 23664 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.834978 21029 raft_consensus.cc:2354] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:19.821282 23668 tablet_copy_client.cc:655] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Starting download of 1 WAL segments...
I20240506 15:15:19.824535 23699 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
W20240506 15:15:19.866111 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 89251.3us
I20240506 15:15:19.872382 23719 raft_consensus.cc:3032] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:19.809094 23676 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20240506 15:15:19.826164 23697 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:19.873638 20878 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 11: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:19.802328 23579 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.876854 20924 leader_election.cc:398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:19.877648 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.878225 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:19.882213 20879 raft_consensus.cc:2379] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:19.795249 23686 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:19.876456 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 54633.1us
I20240506 15:15:19.886821 21027 raft_consensus.cc:1261] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240506 15:15:19.894210 23567 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.897282 20924 leader_election.cc:398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:19.897567 20924 leader_election.cc:401] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240506 15:15:19.897464 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:19.898663 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:19.824231 23642 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:19.891502 23693 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:19.873601 23501 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:19.884886 23587 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:19.962764 23731 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:19.883635 23697 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:19.964905 20955 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 11: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
W20240506 15:15:19.965626 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 61903us
I20240506 15:15:19.972308 20998 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:19.880263 23719 raft_consensus.cc:729] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:19.864715 23577 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20240506 15:15:19.869038 23679 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:19.861584 23656 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:20.004865 23711 log.cc:924] Time spent T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
W20240506 15:15:20.064019 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 165702us
I20240506 15:15:20.065547 20924 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.060982 21685 raft_consensus.cc:2354] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:15:20.011255 23723 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:20.049242 23668 tablet_copy_client.cc:531] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 15:15:20.038285 23731 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:20.037422 23693 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.077272 23694 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:20.099457 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.077188 23723 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:19.982872 23587 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:20.117957 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.120890 20952 raft_consensus.cc:2354] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:20.121080 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.120682 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.121919 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.129851 20958 raft_consensus.cc:2454] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:20.130988 20956 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20240506 15:15:20.122915 23723 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:20.129601 23691 consensus_meta.cc:287] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.001s
I20240506 15:15:20.044054 23699 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:20.117957 23664 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.172942 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.173719 20880 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:20.191869 20846 leader_election.cc:398] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:20.192457 20846 leader_election.cc:401] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240506 15:15:20.132970 23694 raft_consensus.cc:686] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
W20240506 15:15:20.208889 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 44358.1us
I20240506 15:15:20.212407 20880 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:20.168972 23724 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:20.231029 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:20.272367 23726 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20240506 15:15:20.206076 23579 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.277040 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:20.283746 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 111415us
I20240506 15:15:20.315909 20956 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Advancing to term 5
I20240506 15:15:20.244678 20881 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:20.122438 23679 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.328352 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.230713 23642 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.094003 23719 consensus_queue.cc:260] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.180049 23691 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:20.324427 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 425772us
I20240506 15:15:20.233371 23583 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:20.375388 23745 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20240506 15:15:20.376317 20883 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:20.386301 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:20.388083 23734 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
W20240506 15:15:20.388737 23738 log.cc:924] Time spent T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20240506 15:15:20.389791 23712 consensus_queue.cc:1046] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240506 15:15:20.391537 21686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34002 (request call id 149) took 1112 ms. Trace:
I20240506 15:15:20.391958 21686 rpcz_store.cc:276] 0506 15:15:19.268424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.273804 (+ 5380us) service_pool.cc:225] Handling call
0506 15:15:20.364710 (+1090906us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b5400411ed4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:20.380626 (+ 15916us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":719942528}
I20240506 15:15:20.361958 23567 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:20.369160 23694 consensus_queue.cc:237] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.399168 23691 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.362432 23656 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.402112 23656 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:20.403738 23668 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap starting.
I20240506 15:15:20.421788 21030 raft_consensus.cc:2354] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
W20240506 15:15:20.387142 23596 consensus_meta.cc:287] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.693s user 0.010s sys 0.000s
W20240506 15:15:20.404439 23722 log.cc:924] Time spent T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.444s user 0.000s sys 0.001s
W20240506 15:15:20.404732 23736 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.361s user 0.001s sys 0.001s
I20240506 15:15:20.433672 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
W20240506 15:15:20.433629 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47493.3us
W20240506 15:15:20.403803 23618 consensus_meta.cc:287] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.771s user 0.005s sys 0.006s
I20240506 15:15:20.438010 20921 leader_election.cc:398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:20.412333 23695 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.404196 23587 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:20.436443 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 120307us
I20240506 15:15:20.455096 21032 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:20.458233 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:20.461252 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:20.462396 21032 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 11: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:20.465052 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:20.465961 21032 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:20.469678 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:20.470785 21027 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240506 15:15:20.407502 23697 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.438932 23596 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.478611 23697 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.479034 23596 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.485971 20876 raft_consensus.cc:2354] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:20.435333 23585 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:20.490824 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 162582us
I20240506 15:15:20.491267 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.505565 23697 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.506336 23697 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.518812 20956 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:15:20.538905 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.470697 23735 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:20.441995 23618 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.562646 20921 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:20.560818 21032 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:20.514776 23567 raft_consensus.cc:686] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:20.512122 23751 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240506 15:15:20.565788 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.587811 20957 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:20.590786 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:20.536425 23585 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.470961 23695 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:20.612898 23749 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:20.613052 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.622164 20957 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:20.599756 23501 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:20.604036 23697 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.611639 23587 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:20.620677 23583 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:20.631562 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 127999us
W20240506 15:15:20.634716 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44074us
I20240506 15:15:20.635380 20996 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:20.639783 23618 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.635901 23697 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.641985 23742 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20240506 15:15:20.642423 23697 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.643389 23755 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:20.641230 23760 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.650645 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.648638 23583 raft_consensus.cc:2781] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:20.660274 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 1791) took 1021 ms. Trace:
I20240506 15:15:20.660912 20955 rpcz_store.cc:276] 0506 15:15:19.639253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.639392 (+ 139us) service_pool.cc:225] Handling call
0506 15:15:20.660055 (+1020663us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b540032f854. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:20.660193 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731240320}
I20240506 15:15:20.627254 23501 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:20.667500 23753 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20240506 15:15:20.660298 23583 raft_consensus.cc:686] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:20.639849 23757 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.623886 23749 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:20.633639 23587 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.671140 23755 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240506 15:15:20.631140 23567 consensus_queue.cc:237] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.676926 23618 raft_consensus.cc:1050] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
W20240506 15:15:20.677415 23618 raft_consensus.cc:1054] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Unable to promote non-voter 7c8800c6dcaf4b6e8dd58c4149f73c4e: Service unavailable: leader transfer in progress
I20240506 15:15:20.666463 23697 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.679680 23749 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:20.681217 23585 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.681705 23587 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.647711 23723 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.671334 23757 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.686540 23757 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.648816 23760 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:20.687593 21685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34002 (request call id 150) took 1298 ms (client timeout 1630 ms). Trace:
W20240506 15:15:20.688086 21685 rpcz_store.cc:269] 0506 15:15:19.389299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.389419 (+ 120us) service_pool.cc:225] Handling call
0506 15:15:20.687355 (+1297936us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54006dea54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:20.687530 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1295067392}
I20240506 15:15:20.689023 23760 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.689869 23583 consensus_queue.cc:237] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.691416 23501 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:20.664824 23549 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:20.693413 23765 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:20.684643 23723 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:20.692045 23740 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.520s user 0.003s sys 0.000s
I20240506 15:15:20.700208 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.702579 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:20.703384 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 64437.8us
I20240506 15:15:20.704419 20921 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:20.705775 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.706073 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.696431 23695 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:20.734136 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70844.1us
I20240506 15:15:20.734838 23760 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:20.735417 23760 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.739734 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.739953 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.740393 21654 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.742508 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.743170 20958 raft_consensus.cc:3037] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:20.732039 23690 raft_consensus.cc:979] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:20.732110 23695 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:20.750977 20958 raft_consensus.cc:2454] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:20.762172 23549 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:20.741348 23542 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:20.761840 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57492.1us
I20240506 15:15:20.751188 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.765033 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.751894 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.768105 20881 raft_consensus.cc:2354] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9.
I20240506 15:15:20.741431 23632 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:20.769649 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:20.763080 23542 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:20.741312 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:20.747602 23755 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:20.787992 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:20.790658 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.793622 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:20.800659 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.800871 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.802292 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.802548 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:20.803763 20949 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 6: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:20.804865 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.805050 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.807148 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:20.810269 20922 leader_election.cc:398] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10.
I20240506 15:15:20.810765 20922 leader_election.cc:401] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:20.794481 21031 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:20.815761 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:20.821965 21023 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:20.779181 23690 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:20.767930 23695 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.782364 23549 raft_consensus.cc:1067] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Signalling peer 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:20.825614 21024 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:20.820775 23771 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:20.834005 23727 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:20.851300 20922 leader_election.cc:398] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:15:20.859846 20922 leader_election.cc:401] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:20.778582 23632 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
W20240506 15:15:20.854312 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 61535us
W20240506 15:15:20.862334 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49208.2us
I20240506 15:15:20.879000 23582 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:15:20.882882 23542 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:20.888410 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 94739.8us
I20240506 15:15:20.889793 20921 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.885799 20996 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:20.810909 23755 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.900107 21023 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:20.902352 20846 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.904620 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:20.909884 23783 raft_consensus.cc:2781] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:20.910440 23783 raft_consensus.cc:686] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:20.911325 23783 consensus_queue.cc:237] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.906497 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.844971 23690 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:20.865737 23760 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.919986 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:20.926440 20880 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:20.921450 20924 leader_election.cc:398] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9.
I20240506 15:15:20.865116 23727 raft_consensus.cc:686] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:20.924203 23758 raft_consensus.cc:2741] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20240506 15:15:20.933710 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50474 (request call id 1835) took 1172 ms. Trace:
I20240506 15:15:20.903172 23549 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 4. Current term is 5.
I20240506 15:15:20.936554 20883 rpcz_store.cc:276] 0506 15:15:19.754699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.758604 (+ 3905us) service_pool.cc:225] Handling call
0506 15:15:19.758851 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:20.384953 (+626102us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:20.926476 (+541523us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b5400267854. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:20.926742 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1419386240}
I20240506 15:15:20.921955 23656 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:20.924093 23690 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:20.949102 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 80828us
I20240506 15:15:20.950635 21023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:20.951494 20996 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:20.951676 21023 raft_consensus.cc:595] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:20.950140 21024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:47952
I20240506 15:15:20.952147 21023 raft_consensus.cc:603] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:20.952473 20921 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:20.950606 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.031193 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.031880 23773 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421) with higher term. Message: Invalid argument: T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10.
I20240506 15:15:21.032733 23695 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:20.947152 23656 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:20.861513 23771 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.037348 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.023690 23760 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:21.033196 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.049507 20954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b54b354e91941efb28b61b312923d41"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:21.074785 23760 raft_consensus.cc:686] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:21.075258 20954 raft_consensus.cc:595] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
W20240506 15:15:21.049014 23753 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20240506 15:15:21.072492 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 121080us
I20240506 15:15:21.031950 21031 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:21.076275 21026 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:21.033970 23632 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.070353 23730 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.087065 23730 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.087924 23790 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.088497 23790 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:21.089772 23730 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.036432 23727 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.090813 23790 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:21.057497 23785 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.213s user 0.002s sys 0.000s
I20240506 15:15:21.042413 23755 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:21.039837 23741 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.720s user 0.003s sys 0.000s
I20240506 15:15:21.076817 20996 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:21.092027 20876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50474 (request call id 1829) took 1339 ms (client timeout 1555 ms). Trace:
I20240506 15:15:21.072115 23582 tablet_bootstrap.cc:492] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
W20240506 15:15:21.095131 20876 rpcz_store.cc:269] 0506 15:15:19.752893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.759621 (+ 6728us) service_pool.cc:225] Handling call
0506 15:15:21.091823 (+1332202us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400277254. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:21.091950 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668184320}
I20240506 15:15:21.053745 23695 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:21.095818 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.098917 21031 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
W20240506 15:15:21.096846 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 156027us
I20240506 15:15:21.101531 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:21.101697 20924 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.555s (SENT) [suppressed 2 similar messages]
I20240506 15:15:21.078292 23577 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:21.082370 23656 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:21.105274 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.119573 20957 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:21.107584 21031 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:21.108731 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:21.120357 20957 raft_consensus.cc:595] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:21.115063 20846 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:21.115761 20882 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:21.116446 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:21.121510 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.123198 21026 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
W20240506 15:15:21.123716 21026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 419) took 1296 ms (client timeout 1620 ms). Trace:
W20240506 15:15:21.124075 21026 rpcz_store.cc:269] 0506 15:15:19.827806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.037027 (+1209221us) service_pool.cc:225] Handling call
0506 15:15:21.123665 (+ 86638us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:21.124626 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:21.124084 21031 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:21.093202 23771 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:21.143154 20996 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.139632 23733 consensus_queue.cc:1035] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20240506 15:15:21.146837 23744 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:21.147485 23744 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.107950 23585 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240506 15:15:21.149612 21030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1852) took 1386 ms (client timeout 1555 ms). Trace:
I20240506 15:15:21.149739 23744 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:21.142282 23582 ts_tablet_manager.cc:1367] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 1.906s user 0.046s sys 0.026s
W20240506 15:15:21.150071 21030 rpcz_store.cc:269] 0506 15:15:19.762827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.762979 (+ 152us) service_pool.cc:225] Handling call
0506 15:15:21.149406 (+1386427us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b5400139054. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:21.149533 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1508757248}
I20240506 15:15:21.151142 23687 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:21.151767 23687 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.153676 21024 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:21.154201 21024 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:21.166038 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.167667 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.168447 21026 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:21.168640 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.170408 21023 raft_consensus.cc:2354] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:15:21.125576 23632 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:21.135263 23783 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:21.125705 23755 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:21.179008 21024 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:21.181968 20996 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.139981 23577 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:21.162405 23695 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.192873 21026 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:21.195526 23787 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:21.197950 23784 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:21.198719 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.187628 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.198690 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1858) took 1386 ms. Trace:
I20240506 15:15:21.207286 21026 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 11: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:21.171849 23733 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.208920 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.206255 20996 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.191923 23783 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:21.117686 23760 consensus_queue.cc:237] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:21.220808 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 57534.7us
I20240506 15:15:21.186415 23724 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.193564 23583 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:21.221460 23697 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240506 15:15:21.290751 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 99322.4us
I20240506 15:15:21.206668 21028 rpcz_store.cc:276] 0506 15:15:19.803603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.807606 (+ 4003us) service_pool.cc:225] Handling call
0506 15:15:19.822059 (+ 14453us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:19.831716 (+ 9657us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:19.831763 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:19.852820 (+ 21057us) log.cc:842] Serialized 26 byte log entry
0506 15:15:20.074316 (+221496us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:20.074372 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:20.074467 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:21.039083 (+964616us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:21.185251 (+146168us) raft_consensus.cc:1671] finished
0506 15:15:21.185270 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:21.189750 (+ 4480us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27647,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"spinlock_wait_cycles":59594624,"thread_start_us":201235,"threads_started":1,"wal-append.queue_time_us":308450}
I20240506 15:15:21.187121 23632 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.190750 23755 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.360484 20996 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.360832 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:21.286106 23677 raft_consensus.cc:1054] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Unable to promote non-voter 7c8800c6dcaf4b6e8dd58c4149f73c4e: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:21.363109 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 261408us
I20240506 15:15:21.191846 23758 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20240506 15:15:21.287158 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:21.398303 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.399853 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:21.404363 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.405390 20956 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:21.390472 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.246016 23585 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:21.317864 23784 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.415443 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.196075 23787 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.409482 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.411186 20951 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:21.412364 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.419328 20949 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:21.415383 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.290448 23687 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.430696 20924 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:21.225050 23733 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.443266 20845 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.386428 23779 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:21.394119 23697 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.447861 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.398473 23727 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.396844 23658 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.451154 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.396601 23744 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.287544 23577 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.460664 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:21.482760 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.390899 23724 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.372892 23713 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20240506 15:15:21.453868 23585 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.453292 23549 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:21.453117 23541 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:21.498806 21026 raft_consensus.cc:2379] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:21.516656 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:21.517350 21028 raft_consensus.cc:595] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:21.517699 21028 raft_consensus.cc:603] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:21.519690 20996 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:21.438813 23582 raft_consensus.cc:348] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:21.545697 21026 raft_consensus.cc:1226] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20240506 15:15:21.505770 23677 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.457304 23697 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.558290 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:21.557356 20996 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.484490 23658 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.469051 23727 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.418953 23783 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.485647 23680 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240506 15:15:21.437896 23784 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.548827 23582 raft_consensus.cc:729] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:15:21.573212 20996 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.550861 23695 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.455919 23779 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:21.586498 23799 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20240506 15:15:21.519197 23585 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.564564 23687 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:21.627802 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.572014 23746 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.641155 23787 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:21.643580 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 70427.7us
W20240506 15:15:21.632920 23794 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20240506 15:15:21.698257 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.662357 20953 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:21.701529 20953 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:15:21.646147 23632 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.632395 23772 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:21.643121 23746 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:21.682232 23656 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.545s user 0.009s sys 0.001s
W20240506 15:15:21.641774 23807 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20240506 15:15:21.676494 23793 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
I20240506 15:15:21.715832 23656 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.717134 23727 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.720330 23697 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.722322 20958 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:21.719061 23656 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:21.708659 23800 log.cc:924] Time spent T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.297s user 0.002s sys 0.000s
I20240506 15:15:21.724637 23754 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:21.710388 23779 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:21.677076 23712 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.726378 23585 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.641858 23784 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.729792 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.726501 23712 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.714044 23724 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:21.731547 23796 log.cc:924] Time spent T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.261s user 0.001s sys 0.000s
I20240506 15:15:21.730793 20953 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:15:21.724093 23746 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:21.636096 23759 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.258s
I20240506 15:15:21.644057 23749 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:21.759032 21654 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:21.706341 23755 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.737162 23658 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.736122 23772 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:21.771587 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 205527us
I20240506 15:15:21.736654 23712 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:21.775624 20996 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:21.760694 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 310208us
I20240506 15:15:21.645875 23582 consensus_queue.cc:260] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:21.777312 20845 leader_election.cc:398] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:15:21.777854 20845 leader_election.cc:401] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240506 15:15:21.733338 23733 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.767339 23754 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:21.803642 20845 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.813001 23717 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:21.808610 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.789207 23784 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:21.821837 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.788820 23731 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:21.817870 20845 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.827154 23719 raft_consensus.cc:3037] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:21.827478 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1875) took 1039 ms. Trace:
I20240506 15:15:21.801599 23754 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.763522 23749 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.830595 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.827939 21027 rpcz_store.cc:276] 0506 15:15:20.787895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:20.788008 (+ 113us) service_pool.cc:225] Handling call
0506 15:15:20.788326 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:21.315427 (+527101us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b180038ba20. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630df7e75c 00007f630b3a754c 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c1cc5 00007f630e2e7620 00007f630e2e81bb 00007f630e2f29db
0506 15:15:21.317635 (+ 2208us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.317667 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:21.322082 (+ 4415us) log.cc:842] Serialized 26 byte log entry
0506 15:15:21.680711 (+358629us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:21.680872 (+ 161us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:21.681236 (+ 364us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:21.827241 (+146005us) raft_consensus.cc:1671] finished
0506 15:15:21.827259 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:21.827429 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104093,"raft.queue_time_us":56346,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":6055,"spinlock_wait_cycles":7209856,"thread_start_us":302444,"threads_started":2,"wal-append.queue_time_us":406086}
W20240506 15:15:21.832046 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 462038us
I20240506 15:15:21.840189 20922 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:21.827306 23824 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:21.813006 23541 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6.
I20240506 15:15:21.848218 23698 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.813750 23717 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.854987 21653 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:21.863546 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1876) took 1075 ms. Trace:
I20240506 15:15:21.828141 23729 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:21.864104 21029 rpcz_store.cc:276] 0506 15:15:20.788186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:20.788338 (+ 152us) service_pool.cc:225] Handling call
0506 15:15:20.788610 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:20.850665 (+ 62055us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:20.850705 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:20.850744 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:20.850773 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:21.863296 (+1012523us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b54001c20d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:21.863343 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:21.863488 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142279936}
I20240506 15:15:21.869143 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.834829 23754 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:21.844429 20845 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.847371 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:21.877467 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.878167 21029 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:21.878260 21027 raft_consensus.cc:2354] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:21.852824 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.856775 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.859774 20950 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:21.873379 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.835021 23675 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:21.851594 21024 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:21.853688 23808 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.203s user 0.000s sys 0.004s
I20240506 15:15:21.881593 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:21.885895 20950 raft_consensus.cc:3032] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:21.885636 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.886339 20954 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240506 15:15:21.886358 20950 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:21.887805 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:21.853206 23813 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:21.890087 21029 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:21.853590 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:15:21.895793 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 55753.8us
I20240506 15:15:21.888815 20950 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
W20240506 15:15:21.898483 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 54211.5us
I20240506 15:15:21.899260 20950 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:21.899571 20845 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.901505 23733 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.902412 23733 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:21.882052 23699 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:21.901093 20845 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.909709 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:21.915035 21027 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240506 15:15:21.920892 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 106270us
I20240506 15:15:21.923674 20921 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:21.926654 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:21.832350 23731 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.891572 23813 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:21.906177 23759 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:21.888635 23675 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:21.922214 23541 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:21.913334 23724 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:21.929138 20950 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:21.933184 20921 leader_election.cc:398] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:21.939843 20921 leader_election.cc:401] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240506 15:15:21.941252 23821 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.925595 23656 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:21.933948 23582 ts_tablet_manager.cc:1398] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.783s user 0.011s sys 0.001s
I20240506 15:15:21.940771 23724 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.935547 23677 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:21.875967 23729 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.949066 23677 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:21.949512 23677 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:21.925810 23712 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20240506 15:15:21.929829 23699 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:21.940796 23814 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:21.953356 23729 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:21.948333 23582 ts_tablet_manager.cc:1430] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Tablet startup took 797ms
W20240506 15:15:21.957113 23804 log.cc:924] Time spent T e47e3786df1a41928afd8b70061d732a P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.448s user 0.001s sys 0.000s
I20240506 15:15:21.958978 20996 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:21.954425 23823 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240506 15:15:21.935292 23585 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240506 15:15:21.950505 23724 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:21.969075 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.969429 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.970441 23770 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20240506 15:15:21.958271 23680 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6.
I20240506 15:15:21.978374 23743 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:21.979035 23743 raft_consensus.cc:686] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:21.967921 23749 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:21.950872 23656 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:21.983909 23541 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:21.941895 23821 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.012787 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.961761 23699 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:22.011663 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 156753us
I20240506 15:15:22.004937 23541 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:22.051103 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:21.979987 23743 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:22.051880 20956 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:22.059512 23541 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:22.061691 23675 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:22.065598 20996 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:22.085343 23805 log.cc:924] Time spent T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.585s user 0.002s sys 0.000s
I20240506 15:15:22.011644 23717 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:22.050094 23731 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:22.023717 23712 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:22.123726 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:22.124547 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.126179 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.104023 23757 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:22.104642 23746 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:22.114476 23583 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.292191 23813 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:22.290212 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 363669us
I20240506 15:15:22.325945 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.326323 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.326750 20878 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
I20240506 15:15:22.154894 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.127236 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.156716 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.149785 21689 raft_consensus.cc:1203] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20240506 15:15:22.322875 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.327692 20879 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:22.328104 20877 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:22.110762 23695 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:22.341192 21027 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:22.343612 23694 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:22.344229 23694 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:22.303231 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.331884 20998 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:22.406766 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:22.414322 21024 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
W20240506 15:15:22.419644 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 87422.6us
I20240506 15:15:22.419932 21656 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:22.155683 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.354955 20875 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240506 15:15:22.415740 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 81900.6us
W20240506 15:15:22.288689 23728 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 2.394s user 0.013s sys 0.010s
I20240506 15:15:22.433094 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:22.155812 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:22.345779 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:22.296270 23684 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.275141 23814 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:22.181593 23668 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240506 15:15:22.482056 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:22.330314 23730 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.406926 23693 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:22.343003 23754 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:22.487732 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 67290.5us
I20240506 15:15:22.485627 23699 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.118479 23712 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.287642 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:22.614634 21686 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:15:22.414666 23757 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.341121 23824 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.497684 23501 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:22.177973 23733 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
W20240506 15:15:21.957043 23582 ts_tablet_manager.cc:1432] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Trace:
0506 15:15:18.314661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:18.314821 (+ 160us) service_pool.cc:225] Handling call
0506 15:15:18.315925 (+ 1104us) ts_tablet_manager.cc:903] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Initiating tablet copy from peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
0506 15:15:18.603312 (+287387us) tablet_copy_client.cc:361] Tablet copy session begun
0506 15:15:18.614377 (+ 11065us) tablet_metadata.cc:641] Metadata flushed
0506 15:15:18.614613 (+ 236us) tablet_copy_client.cc:490] Wrote new tablet metadata
0506 15:15:19.010708 (+396095us) tablet_copy_client.cc:411] Wrote new consensus metadata
0506 15:15:19.011117 (+ 409us) tablet_replica.cc:205] Creating consensus instance
0506 15:15:19.011522 (+ 405us) ts_tablet_manager.cc:958] Replica registered.
0506 15:15:19.011602 (+ 80us) inbound_call.cc:173] Queueing success response
0506 15:15:19.236005 (+224403us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0506 15:15:21.150833 (+1914828us) ts_tablet_manager.cc:1399] Starting tablet replica
0506 15:15:21.151390 (+ 557us) tablet_replica.cc:259] Starting instrumentation
0506 15:15:21.151956 (+ 566us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":187,"mutex_wait_us":24996,"raft.queue_time_us":8541,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":9606,"tablet-copy.queue_time_us":166,"tablet-download-pool.queue_time_us":5454,"tablet-download-pool.run_cpu_time_us":1087,"tablet-download-pool.run_wall_time_us":146808,"thread_start_us":70792,"threads_started":1}
I20240506 15:15:22.259236 23821 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.341850 23695 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.497627 23694 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:22.485999 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 46806.6us
I20240506 15:15:22.316310 23656 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.484606 20882 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:22.602314 20950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.026s sys 0.027s
W20240506 15:15:22.483650 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 63830.1us
I20240506 15:15:22.540705 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:22.414245 23746 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.340752 23749 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.196463 23677 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.421281 23583 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.606395 21024 raft_consensus.cc:2362] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10. Re-sending same reply.
I20240506 15:15:22.406478 23813 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:22.610592 23832 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:22.608292 23730 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.629388 23501 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:22.613231 23834 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.137s
I20240506 15:15:22.612646 23831 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.606575 23814 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.609742 23693 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.652930 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1890) took 1294 ms. Trace:
I20240506 15:15:22.674597 21031 rpcz_store.cc:276] 0506 15:15:21.358219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.889792 (+531573us) service_pool.cc:225] Handling call
0506 15:15:21.890052 (+ 260us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:21.890412 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.890440 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:21.890964 (+ 524us) log.cc:842] Serialized 26 byte log entry
0506 15:15:22.098863 (+207899us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:22.098914 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.099285 (+ 371us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:22.652703 (+553418us) raft_consensus.cc:1671] finished
0506 15:15:22.652723 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:22.652841 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":997,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":11338,"thread_start_us":184340,"threads_started":1,"wal-append.queue_time_us":708641}
I20240506 15:15:22.677946 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1893) took 1106 ms. Trace:
W20240506 15:15:22.668920 23839 log.cc:924] Time spent T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
W20240506 15:15:22.638675 23763 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.086s user 0.036s sys 0.105s
I20240506 15:15:22.613009 23684 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.605681 23830 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.687134 23830 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.687495 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:22.688377 23684 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.689211 21685 raft_consensus.cc:1261] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:22.689288 23830 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.668511 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1534) took 3103 ms. Trace:
I20240506 15:15:22.690140 21025 rpcz_store.cc:276] 0506 15:15:19.560554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.562411 (+ 1857us) service_pool.cc:225] Handling call
0506 15:15:19.579282 (+ 16871us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:19.581531 (+ 2249us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:19.581560 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:19.600304 (+ 18744us) log.cc:842] Serialized 26 byte log entry
0506 15:15:19.735053 (+134749us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:19.735108 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:19.735200 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:22.067305 (+2332105us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:22.652455 (+585150us) raft_consensus.cc:1671] finished
0506 15:15:22.652481 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:22.664060 (+ 11579us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9154,"raft.queue_time_us":2358,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":21185,"spinlock_wait_cycles":12016896,"thread_start_us":83350,"threads_started":1,"wal-append.queue_time_us":286309}
I20240506 15:15:22.693624 23677 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.694244 23730 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.670725 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:22.697026 21028 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:22.697500 23583 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:22.697700 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1895) took 1036 ms. Trace:
I20240506 15:15:22.698175 21028 rpcz_store.cc:276] 0506 15:15:21.661784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.696447 (+1034663us) service_pool.cc:225] Handling call
0506 15:15:22.696820 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.697428 (+ 608us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.697615 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:22.699714 23693 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.700667 21690 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
W20240506 15:15:22.701996 23541 consensus_meta.cc:287] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.584s user 0.009s sys 0.002s
I20240506 15:15:22.645224 23695 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:22.623263 23824 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:22.665395 23749 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.705102 23824 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:22.705619 23844 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:22.706382 23749 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:22.674077 20954 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 8: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:22.678406 21031 rpcz_store.cc:276] 0506 15:15:21.571783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.676890 (+1105107us) service_pool.cc:225] Handling call
0506 15:15:22.677271 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.677513 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:22.677550 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:22.677607 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:15:22.677652 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.677721 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:22.677872 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 15:15:22.673240 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64161.5us
W20240506 15:15:22.653246 20950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.026s sys 0.027s
I20240506 15:15:22.702600 23541 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.709764 20922 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:22.712635 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:22.710803 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1820) took 1299 ms. Trace:
I20240506 15:15:22.713891 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1898) took 1017 ms. Trace:
I20240506 15:15:22.713654 20955 rpcz_store.cc:276] 0506 15:15:21.411481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.705851 (+1294370us) service_pool.cc:225] Handling call
0506 15:15:22.706167 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.710396 (+ 4229us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:22.710434 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:22.710489 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:22.710534 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.710599 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:22.710740 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:22.689106 23757 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.714550 21031 rpcz_store.cc:276] 0506 15:15:21.696416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.707685 (+1011269us) service_pool.cc:225] Handling call
0506 15:15:22.707981 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.708141 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:22.708165 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:22.708202 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:22.708230 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.713728 (+ 5498us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:22.713834 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:22.715310 23814 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.707599 20876 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 11.
I20240506 15:15:22.716143 23746 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.717475 23677 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:22.684659 23831 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.721357 23831 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.722112 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:22.722898 20949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 1821) took 1307 ms. Trace:
I20240506 15:15:22.723237 20949 rpcz_store.cc:276] 0506 15:15:21.415951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.721833 (+1305882us) service_pool.cc:225] Handling call
0506 15:15:22.722839 (+ 1006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:22.726341 23501 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:22.727262 23719 consensus_meta.cc:287] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.899s user 0.010s sys 0.003s
I20240506 15:15:22.727823 23777 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:22.728384 23693 consensus_queue.cc:1035] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:22.707135 20958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35816 (request call id 1511) took 1314 ms. Trace:
I20240506 15:15:22.732882 23684 consensus_queue.cc:1046] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:22.732743 20958 rpcz_store.cc:276] 0506 15:15:21.392995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.395232 (+ 2237us) service_pool.cc:225] Handling call
0506 15:15:21.395451 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:21.723385 (+327934us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.706893 (+983508us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b540033f9d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:22.707057 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":750883712}
I20240506 15:15:22.733971 23684 raft_consensus.cc:3032] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:22.734400 23684 raft_consensus.cc:729] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:22.733130 21031 raft_consensus.cc:1261] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240506 15:15:22.735128 23684 consensus_queue.cc:260] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.736706 23695 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.740497 23695 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:22.740918 23754 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.738754 21031 raft_consensus.cc:1261] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:22.741760 23677 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:22.742575 23712 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.674453 23832 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.743808 23832 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Advancing to term 11
W20240506 15:15:22.710283 23763 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.159s user 0.036s sys 0.106s
I20240506 15:15:22.659324 23733 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:22.752470 23684 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:22.752606 23733 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:22.755427 23541 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.756439 20878 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:22.760354 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.759882 20882 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:22.762952 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.763059 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:22.767675 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:22.765827 20882 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 4.
W20240506 15:15:22.775723 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 111961us
W20240506 15:15:22.783582 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 115124us
I20240506 15:15:22.784830 20845 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:22.728400 23777 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.787029 20883 raft_consensus.cc:1261] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240506 15:15:22.787403 20883 pending_rounds.cc:76] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 4
I20240506 15:15:22.787611 20883 pending_rounds.cc:97] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240506 15:15:22.787897 20883 raft_consensus.cc:2866] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:22.789708 20883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 1800) took 1392 ms. Trace:
I20240506 15:15:22.790717 20882 raft_consensus.cc:1261] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240506 15:15:22.790136 20883 rpcz_store.cc:276] 0506 15:15:21.397807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.399448 (+ 1641us) service_pool.cc:225] Handling call
0506 15:15:21.440473 (+ 41025us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.788549 (+1348076us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.789448 (+ 899us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000bbb54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:22.789624 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3095816064}
I20240506 15:15:22.796202 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.799913 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.800496 23668 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 15:15:22.806617 23668 tablet_bootstrap.cc:492] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Bootstrap complete.
I20240506 15:15:22.807428 23668 ts_tablet_manager.cc:1367] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent bootstrapping tablet: real 2.404s user 0.059s sys 0.044s
I20240506 15:15:22.799341 23834 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:22.810636 23834 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.811091 23834 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:22.813339 20878 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:22.813931 20878 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:22.817560 20881 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20240506 15:15:22.814728 20878 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.820664 20878 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:22.826058 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.830753 20881 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:22.891597 23813 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:22.892516 23813 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.895290 23677 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.896020 23677 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.897507 23567 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.898244 23567 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.899243 23693 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.899961 23693 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.900843 23567 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.902419 23693 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.904400 23577 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.905156 23577 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:22.908736 23693 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.910323 23693 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:22.913002 23677 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.915141 23813 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.908006 23577 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.918715 23693 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:22.975056 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:22.975363 23851 log.cc:924] Time spent T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20240506 15:15:22.983680 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.989883 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:22.992296 23668 raft_consensus.cc:348] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:22.993399 23668 raft_consensus.cc:729] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Initialized, Role: LEARNER
I20240506 15:15:22.990020 23642 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:22.994401 23642 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:22.999061 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.000705 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:22.971729 23693 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:22.994185 23668 consensus_queue.cc:260] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:22.998090 23832 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:23.005559 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.030031 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1543) took 1866 ms. Trace:
I20240506 15:15:23.059058 21023 rpcz_store.cc:276] 0506 15:15:21.163555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.182731 (+ 19176us) service_pool.cc:225] Handling call
0506 15:15:21.183115 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:21.455557 (+272442us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.455588 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:21.455627 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:21.455657 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.029718 (+1574061us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b54001e2654. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:23.029783 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.029966 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626030976}
I20240506 15:15:23.060550 20874 raft_consensus.cc:2396] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20240506 15:15:23.063227 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.064688 20953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 320) took 1335 ms. Trace:
I20240506 15:15:22.998792 23642 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.069675 21025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1894) took 1456 ms. Trace:
I20240506 15:15:23.071110 21025 rpcz_store.cc:276] 0506 15:15:21.613710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.691938 (+1078228us) service_pool.cc:225] Handling call
0506 15:15:22.692298 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.711867 (+ 19569us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:22.711913 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:22.711968 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:22.712014 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.069417 (+357403us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b54001171d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:23.069488 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.069616 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44542336}
I20240506 15:15:23.074887 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.066570 23693 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.075784 21032 raft_consensus.cc:1226] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:23.065099 20953 rpcz_store.cc:276] 0506 15:15:21.729262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.733478 (+ 4216us) service_pool.cc:225] Handling call
0506 15:15:21.733909 (+ 431us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:21.734326 (+ 417us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.734367 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:21.764513 (+ 30146us) log.cc:842] Serialized 26 byte log entry
0506 15:15:22.728116 (+963603us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b18002945e0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3a6eb5 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f630df704eb 00007f630df75f93 00007f630df76341 00007f630e2c381f
0506 15:15:22.784040 (+ 55924us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:22.784087 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:22.784177 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:23.059928 (+275751us) raft_consensus.cc:1671] finished
0506 15:15:23.059945 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.064207 (+ 4262us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6959,"raft.queue_time_us":15803,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":5519,"spinlock_wait_cycles":2995456,"thread_start_us":55720,"threads_started":1,"wal-append.queue_time_us":1014898}
I20240506 15:15:23.079355 20878 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240506 15:15:23.080158 20878 pending_rounds.cc:76] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 4
I20240506 15:15:23.082105 21025 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
W20240506 15:15:23.086685 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 98930us
I20240506 15:15:23.086714 21032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:47936 (request call id 621) took 1968 ms. Trace:
I20240506 15:15:23.083963 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1556) took 1684 ms. Trace:
I20240506 15:15:23.084985 21684 raft_consensus.cc:1226] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 10. Current term is 11. Ops: []
I20240506 15:15:23.085971 23777 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.090273 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.070585 23668 ts_tablet_manager.cc:1398] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent starting tablet: real 0.260s user 0.006s sys 0.001s
I20240506 15:15:23.087705 21029 rpcz_store.cc:276] 0506 15:15:21.399961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.897023 (+497062us) service_pool.cc:225] Handling call
0506 15:15:21.897390 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:21.966687 (+ 69297us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.966733 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:21.967205 (+ 472us) log.cc:842] Serialized 26 byte log entry
0506 15:15:22.181178 (+213973us) raft_consensus.cc:1631] Marking committed up to 7
0506 15:15:22.315596 (+134418us) log.cc:842] Serialized 14 byte log entry
0506 15:15:22.414798 (+ 99202us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.083407 (+668609us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b54001c2fd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:23.083479 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:23.083560 (+ 81us) raft_consensus.cc:1671] finished
0506 15:15:23.083695 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.083893 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116638,"prepare.queue_time_us":58924,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":73,"raft.queue_time_us":113898,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":17618,"spinlock_wait_cycles":123825536,"thread_start_us":193307,"threads_started":1,"wal-append.queue_time_us":632250}
I20240506 15:15:23.087561 21032 rpcz_store.cc:276] 0506 15:15:21.118006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.085641 (+1967635us) service_pool.cc:225] Handling call
0506 15:15:23.086651 (+ 1010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 15:15:23.087734 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.092803 20892 tablet_copy_service.cc:342] P 00fe71c1675441379f5f2c7490f9e63a: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-6d096daea6fa4799a608705b3dc732db received from {username='slave'} at 127.0.0.1:49818
I20240506 15:15:23.093128 20892 tablet_copy_service.cc:433] P 00fe71c1675441379f5f2c7490f9e63a: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-6d096daea6fa4799a608705b3dc732db on tablet 6d096daea6fa4799a608705b3dc732db with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.093001 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.088811 20878 pending_rounds.cc:97] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240506 15:15:23.088696 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.088997 20954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35810 (request call id 1837) took 1118 ms. Trace:
I20240506 15:15:23.095597 20878 raft_consensus.cc:2866] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:23.095821 20954 rpcz_store.cc:276] 0506 15:15:21.968937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.969130 (+ 193us) service_pool.cc:225] Handling call
0506 15:15:23.086651 (+1117521us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54002b5554. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:23.086795 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1612654976}
I20240506 15:15:23.100594 21654 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.103240 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.100088 23795 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:23.102933 23618 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:23.104748 23848 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.104988 23795 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.105147 23618 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.100827 23770 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:23.108372 23693 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.104074 20878 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:23.111917 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.110592 23770 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.113590 20877 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:23.108374 23618 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.116019 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.121026 23770 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.121624 20957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1843) took 1018 ms. Trace:
I20240506 15:15:23.122063 20957 rpcz_store.cc:276] 0506 15:15:22.103223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.105161 (+ 1938us) service_pool.cc:225] Handling call
0506 15:15:22.105591 (+ 430us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.636000 (+530409us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:22.636045 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:22.636105 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:22.636151 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.121367 (+485216us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b540033efd4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:23.121437 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.121565 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1219164544}
W20240506 15:15:23.123193 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 788861us
I20240506 15:15:23.108844 23795 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.139793 21026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1889) took 1781 ms. Trace:
I20240506 15:15:23.143816 21026 rpcz_store.cc:276] 0506 15:15:21.358053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:21.562687 (+204634us) service_pool.cc:225] Handling call
0506 15:15:21.564975 (+ 2288us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:21.632843 (+ 67868us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:21.632892 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:21.632956 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:21.633002 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.139428 (+1506426us) spinlock_profiling.cc:243] Waited 67.6 ms on lock 0x7b54001ae354. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:23.139497 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.139702 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155369856}
I20240506 15:15:23.152511 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.154049 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49620 (request call id 425) took 1584 ms. Trace:
I20240506 15:15:23.157864 20957 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:23.154467 21024 rpcz_store.cc:276] 0506 15:15:21.569905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.668355 (+1098450us) service_pool.cc:225] Handling call
0506 15:15:22.668727 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:23.153707 (+484980us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:23.153735 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:23.153774 (+ 39us) raft_consensus.cc:1631] Marking committed up to 8
0506 15:15:23.153802 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.153898 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.154000 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":484604}
I20240506 15:15:23.165050 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.166301 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:23.168876 21030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1896) took 1491 ms. Trace:
I20240506 15:15:23.171483 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.166457 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.173272 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.151487 23743 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.171557 21027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1907) took 1161 ms. Trace:
I20240506 15:15:23.169250 21030 rpcz_store.cc:276] 0506 15:15:21.677068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.699754 (+1022686us) service_pool.cc:225] Handling call
0506 15:15:22.700084 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.711444 (+ 11360us) raft_consensus.cc:1520] Early marking committed up to index 5
0506 15:15:22.716826 (+ 5382us) log.cc:842] Serialized 14 byte log entry
0506 15:15:23.168565 (+451739us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:23.168610 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:23.168659 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.168722 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.168823 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":384109,"prepare.queue_time_us":6759,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"thread_start_us":67290,"threads_started":1,"wal-append.queue_time_us":60958}
W20240506 15:15:23.178397 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 847618us
I20240506 15:15:23.188596 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.154392 23813 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.180775 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.180160 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.183800 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:23.195529 20950 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:23.185357 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.191603 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.180042 20951 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:23.159626 23689 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:23.201058 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.166312 23676 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.208827 23676 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.209453 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.210832 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.211381 20998 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.164091 23749 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.213013 20998 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.217900 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.220829 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.114850 23834 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.228404 23860 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.228993 23860 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.230235 23834 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.231153 23860 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.232971 21688 raft_consensus.cc:1203] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Deduplicated request from leader. Original: 7.5->[7.6-7.6] Dedup: 7.6->[]
I20240506 15:15:23.236627 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.237314 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.238061 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.238828 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:23.239183 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.240636 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.241438 21026 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:23.241708 21029 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:23.242730 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.194944 23743 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.245174 20778 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54782 (request call id 213) took 3844 ms. Trace:
I20240506 15:15:23.245507 21032 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:23.246423 23743 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.217859 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.249104 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.249338 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.224449 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.221825 23744 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.227326 23613 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:23.217422 23689 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.247836 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:23.240206 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:23.262594 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 49638.5us
I20240506 15:15:23.211481 23676 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.164389 23821 consensus_queue.cc:1035] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20240506 15:15:23.272972 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.291599 21026 raft_consensus.cc:1226] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:23.274309 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.275660 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:23.279399 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:23.239131 23852 log.cc:924] Time spent T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.285s user 0.000s sys 0.002s
I20240506 15:15:23.282078 20955 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:23.274580 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.293535 20845 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:23.290663 20950 raft_consensus.cc:2454] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:23.299211 21042 tablet_copy_service.cc:342] P 7610624a16b44896b6aacea1928ab20e: Request end of tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-a9f4d07ea30646d0805b63bb641358be received from {username='slave'} at 127.0.0.1:49620
I20240506 15:15:23.222504 23832 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 11 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.271740 23549 raft_consensus.cc:979] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:23.300190 21028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1899) took 1597 ms. Trace:
I20240506 15:15:23.308995 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.305490 21028 rpcz_store.cc:276] 0506 15:15:21.702763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.712244 (+1009481us) service_pool.cc:225] Handling call
0506 15:15:22.731148 (+ 18904us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:22.731488 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 4
0506 15:15:22.733649 (+ 2161us) log.cc:842] Serialized 14 byte log entry
0506 15:15:23.170448 (+436799us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:23.177461 (+ 7013us) log.cc:842] Serialized 564 byte log entry
0506 15:15:23.177607 (+ 146us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:23.177639 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.177708 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:23.238612 (+ 60904us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b44000232cc. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b2d9685 00007f630b2da0c2 00007f630df82441 00007f630df7dd4a 00007f630df71b1e 00007f630df70e66 00007f630df70a88 00007f630df7a481 00007f630df7a439 00007f630df7a3c9
0506 15:15:23.299919 (+ 61307us) raft_consensus.cc:1671] finished
0506 15:15:23.300016 (+ 97us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.300138 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":385324,"prepare.queue_time_us":1765,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":387913,"spinlock_wait_cycles":21424384,"thread_start_us":51137,"threads_started":1,"wal-append.queue_time_us":50715}
I20240506 15:15:23.299685 21042 tablet_copy_service.cc:433] P 7610624a16b44896b6aacea1928ab20e: ending tablet copy session 7c8800c6dcaf4b6e8dd58c4149f73c4e-a9f4d07ea30646d0805b63bb641358be on tablet a9f4d07ea30646d0805b63bb641358be with peer 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.162130 23848 consensus_queue.cc:1035] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.130s
I20240506 15:15:23.272541 23749 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:23.291221 23735 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.278806 23613 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.295320 23698 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.332206 20921 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:23.333732 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.245671 20778 rpcz_store.cc:276] 0506 15:15:19.401233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.401356 (+ 123us) service_pool.cc:225] Handling call
0506 15:15:23.245073 (+3843717us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:19.634726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:19.812688 (+177962us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:19.815652 (+ 2964us) write_op.cc:435] Acquired schema lock
0506 15:15:19.817794 (+ 2142us) tablet.cc:660] Decoding operations
0506 15:15:19.876039 (+ 58245us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:19.884763 (+ 8724us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:19.895490 (+ 10727us) tablet.cc:683] Acquiring locks for 1 operations
0506 15:15:19.958821 (+ 63331us) tablet.cc:699] Row locks acquired
0506 15:15:19.968477 (+ 9656us) write_op.cc:260] PREPARE: finished
0506 15:15:20.010251 (+ 41774us) write_op.cc:270] Start()
0506 15:15:20.038434 (+ 28183us) write_op.cc:276] Timestamp: P: 1715008520008866 usec, L: 0
0506 15:15:20.043552 (+ 5118us) op_driver.cc:331] REPLICATION: starting
0506 15:15:20.169528 (+125976us) log.cc:842] Serialized 541 byte log entry
0506 15:15:20.881706 (+712178us) op_driver.cc:444] REPLICATION: finished
0506 15:15:20.916137 (+ 34431us) write_op.cc:301] APPLY: starting
0506 15:15:20.916340 (+ 203us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:20.916465 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:20.916651 (+ 186us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:20.916922 (+ 271us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:20.917150 (+ 228us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 15:15:20.917217 (+ 67us) write_op.cc:312] APPLY: finished
0506 15:15:20.938362 (+ 21145us) log.cc:842] Serialized 23 byte log entry
0506 15:15:20.949701 (+ 11339us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:20.958301 (+ 8600us) write_op.cc:454] Released schema lock
0506 15:15:21.027606 (+ 69305us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:21.097298 (+ 69692us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:21.106286 (+ 8988us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":30883,"apply.run_cpu_time_us":12026,"apply.run_wall_time_us":211416,"num_ops":1,"prepare.queue_time_us":116171,"prepare.run_cpu_time_us":21670,"prepare.run_wall_time_us":851657,"raft.queue_time_us":117192,"raft.run_cpu_time_us":10321,"raft.run_wall_time_us":60850,"replication_time_us":806707,"thread_start_us":359024,"threads_started":4,"wal-append.queue_time_us":262518,"wal-append.run_cpu_time_us":8480,"wal-append.run_wall_time_us":1604777}]]}
I20240506 15:15:23.336453 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.312441 23583 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:23.301859 23834 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.337518 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:23.313603 23684 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:23.339212 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.345525 20878 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 11.
I20240506 15:15:23.339521 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.213135 23813 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:23.326175 20846 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.351300 23689 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.352813 23692 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:23.353443 23692 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:23.349057 23684 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.355489 23693 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index 3 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:23.356727 23684 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:23.335678 23679 consensus_queue.cc:1046] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20240506 15:15:23.360055 23679 raft_consensus.cc:3032] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:23.345419 23795 raft_consensus.cc:2741] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240506 15:15:23.360460 23679 raft_consensus.cc:729] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:23.361330 23679 consensus_queue.cc:260] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.361852 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.362509 23679 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:23.362643 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.363327 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:23.320722 23749 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.313475 23584 consensus_queue.cc:1046] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20240506 15:15:23.341248 23691 consensus_queue.cc:1046] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240506 15:15:23.365017 23584 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.365290 23691 raft_consensus.cc:3032] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:23.317778 23811 consensus_queue.cc:1046] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20240506 15:15:23.365732 23691 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:23.365515 23584 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.366240 23811 raft_consensus.cc:3032] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:23.366587 23811 raft_consensus.cc:729] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:23.367542 23584 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.367442 23811 consensus_queue.cc:260] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.336875 23613 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:23.368888 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.369959 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.370501 21030 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:23.371234 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.313338 23744 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20240506 15:15:23.319646 23549 raft_consensus.cc:1067] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:23.368101 23814 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:23.319831 23587 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20240506 15:15:23.377506 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.364431 23749 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:23.365033 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.379535 21026 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:23.320225 23780 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240506 15:15:23.385653 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.372220 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.387077 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.387408 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.387864 23813 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.388087 21030 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:23.329885 23735 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.388953 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.390017 23735 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.391099 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.392154 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.392725 21025 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:23.375685 23584 raft_consensus.cc:3032] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:23.393386 23584 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:23.394176 23584 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.313673 23743 consensus_queue.cc:1035] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20240506 15:15:23.395692 21025 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:23.388317 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.374542 23755 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.402529 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.404654 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:23.400158 23811 raft_consensus.cc:3037] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:23.406438 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.406646 23684 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.407711 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:23.408583 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:23.409076 20950 raft_consensus.cc:1226] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:23.402092 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.409862 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.187114 21027 rpcz_store.cc:276] 0506 15:15:22.010071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.808338 (+798267us) service_pool.cc:225] Handling call
0506 15:15:22.808621 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:22.808828 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 5
0506 15:15:22.809150 (+ 322us) log.cc:842] Serialized 14 byte log entry
0506 15:15:23.171264 (+362114us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0506 15:15:23.171319 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:23.171358 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.171413 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.171507 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":361904,"prepare.queue_time_us":371,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":375769}
I20240506 15:15:23.411768 20875 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:23.389958 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.412324 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:23.415025 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:23.366541 23691 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.435544 23771 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:23.435820 23684 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.436022 23771 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:23.436535 23771 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:23.341383 23698 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.440564 23549 raft_consensus.cc:482] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.355508 23692 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:23.459858 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:23.385656 23780 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.441608 23765 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:23.472399 20924 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:23.441092 23549 raft_consensus.cc:504] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:23.374089 23834 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:23.402601 23755 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.489288 23759 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:23.494907 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 337484us
I20240506 15:15:23.273190 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:23.526731 23613 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.532361 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 85) took 4087 ms. Trace:
I20240506 15:15:23.568697 20779 rpcz_store.cc:276] 0506 15:15:19.444814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.445005 (+ 191us) service_pool.cc:225] Handling call
0506 15:15:23.532241 (+4087236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9914,"reported_tablets":3}
I20240506 15:15:23.534190 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:23.570008 20875 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:23.573727 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:23.544342 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 85012.7us
I20240506 15:15:23.578212 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.579299 21653 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.545061 20996 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.581239 21653 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:23.544435 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 72175.9us
I20240506 15:15:23.500936 23698 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.582978 23861 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.532745 23784 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.551990 23780 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:23.535733 23676 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.606617 23676 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.609308 23676 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.610324 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1563) took 1387 ms. Trace:
I20240506 15:15:23.533610 23813 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.610692 21023 rpcz_store.cc:276] 0506 15:15:22.223051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.066665 (+843614us) service_pool.cc:225] Handling call
0506 15:15:23.066966 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:23.188740 (+121774us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:23.188770 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:23.190150 (+ 1380us) log.cc:842] Serialized 26 byte log entry
0506 15:15:23.508728 (+318578us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:23.508797 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.508898 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:23.610114 (+101216us) raft_consensus.cc:1671] finished
0506 15:15:23.610134 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.610268 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":130265,"raft.queue_time_us":108872,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":19169,"spinlock_wait_cycles":1892224,"thread_start_us":257730,"threads_started":1,"wal-append.queue_time_us":386793}
I20240506 15:15:23.612514 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.550397 23765 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:23.556001 23767 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:23.590732 20958 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:23.587289 23763 consensus_queue.cc:1046] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20240506 15:15:23.580358 23759 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.595767 20996 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.596405 23809 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240506 15:15:23.598081 23834 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:23.560617 23803 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240506 15:15:23.599915 23755 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.599987 23861 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:23.601619 23549 leader_election.cc:288] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:23.604647 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.643242 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.583304 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.622522 23767 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.625169 20921 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:23.585180 23721 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:23.631453 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.584762 23795 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.634696 23803 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:23.633615 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 46617.8us
I20240506 15:15:23.612553 23784 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.615270 21030 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index 3 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:23.640195 23613 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.623135 23765 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
W20240506 15:15:23.634817 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 53645.3us
I20240506 15:15:23.622721 23813 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.618502 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.638648 20958 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:23.652961 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.635696 23585 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:23.657908 20921 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:23.641044 23698 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.638528 23835 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:23.659036 23692 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:23.624445 23587 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.699412 23587 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.701668 23587 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.633056 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:23.702754 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 69807.6us
I20240506 15:15:23.703135 23784 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:23.678800 20998 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:23.644722 23834 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.652123 23767 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:23.686169 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.649916 23549 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.696058 21031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1906) took 1718 ms. Trace:
I20240506 15:15:23.697163 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b540a7832fba49b5b2623616240c2711" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:23.698848 23692 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:23.703702 23613 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:23.704207 23689 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:23.678340 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:23.705611 20996 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.707930 23691 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
W20240506 15:15:23.721576 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 63756.7us
I20240506 15:15:23.669054 23803 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:23.670807 23795 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:23.727008 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 48353.6us
I20240506 15:15:23.789433 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.748119 23692 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:23.726188 21031 rpcz_store.cc:276] 0506 15:15:21.977628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:22.742725 (+765097us) service_pool.cc:225] Handling call
0506 15:15:22.743881 (+ 1156us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:23.167494 (+423613us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:23.167530 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:23.168045 (+ 515us) log.cc:842] Serialized 26 byte log entry
0506 15:15:23.524010 (+355965us) raft_consensus.cc:1631] Marking committed up to 3
0506 15:15:23.524070 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:23.524181 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:23.695838 (+171657us) raft_consensus.cc:1671] finished
0506 15:15:23.695860 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:23.695996 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":440235,"raft.queue_time_us":427879,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":18372,"thread_start_us":355501,"threads_started":1,"wal-append.queue_time_us":466754}
I20240506 15:15:23.784222 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.712663 23698 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:23.794167 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 88639.4us
I20240506 15:15:23.728232 23834 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.827805 20996 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:23.793144 23784 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:23.793247 23857 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:23.840106 20996 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:23.800472 23662 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:23.795704 23758 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240506 15:15:23.793345 23549 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.852089 23549 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.846433 23661 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.853188 23549 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:23.854332 23698 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.855754 23549 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:23.859373 23749 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.860603 21029 raft_consensus.cc:2379] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 7: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:23.861455 23813 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.863670 23692 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.864504 23765 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.865818 23692 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.867048 23613 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.868340 23857 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:23.870776 23857 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.837810 23587 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:23.875268 23679 consensus_meta.cc:287] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.512s user 0.014s sys 0.000s
I20240506 15:15:23.875082 23587 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:23.876374 23869 log.cc:924] Time spent T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20240506 15:15:23.877746 23834 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.839031 23784 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:23.832857 23811 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:23.887753 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 47746.5us
W20240506 15:15:23.890381 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70795.7us
I20240506 15:15:23.891713 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:23.897810 21032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.022s sys 0.048s
W20240506 15:15:23.914397 21032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.023s sys 0.048s
I20240506 15:15:23.848029 23758 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:23.916611 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.923189 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:23.925024 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 342297us
I20240506 15:15:23.918475 20881 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 3.
I20240506 15:15:23.933362 20878 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:23.933805 20924 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:23.935317 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.902426 23809 raft_consensus.cc:2781] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:23.912031 23784 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:23.944636 21690 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:23.918010 23583 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:23.949878 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:23.951957 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:23.943230 21032 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:23.956665 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:23.932396 23758 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:23.943948 23618 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:23.940980 23759 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.903395 23549 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:23.957312 23809 raft_consensus.cc:686] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:23.972039 23662 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:23.970778 23585 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20240506 15:15:23.948912 23723 consensus_queue.cc:1035] T c12c136a9dc4474897d6b8a669145139 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240506 15:15:24.003407 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:23.982678 23765 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:23.958209 23780 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:23.976156 23811 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20240506 15:15:23.965322 23583 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.060952 21690 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEARNER]: Advancing to term 7
I20240506 15:15:24.001897 23662 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
W20240506 15:15:24.165493 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 215692us
I20240506 15:15:24.172299 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.173916 20924 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.176115 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:15:24.152806 23771 consensus_meta.cc:287] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.716s user 0.011s sys 0.005s
I20240506 15:15:24.064608 23692 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:24.154850 23662 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
W20240506 15:15:24.189272 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 57035us
I20240506 15:15:24.171998 23877 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:24.165786 23859 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:24.154367 23749 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.207604 20924 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.170989 23587 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:24.171983 23585 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:24.213759 20924 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.215451 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.227865 23837 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:24.192751 23771 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.228371 23837 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:24.229224 23837 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.229933 23771 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.230315 23837 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.230535 23771 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.231132 23771 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:24.232105 23873 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:24.232642 23873 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.234649 23873 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.235148 23873 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.235342 23662 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.237692 23662 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.200912 23809 consensus_queue.cc:237] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.204929 23859 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:24.244961 23859 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.246407 23859 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:24.211701 23877 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.250959 23877 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:24.211411 23618 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:24.212568 23692 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.221184 23585 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.274986 23585 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.270424 23786 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.276924 23786 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.278950 23784 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.279577 23786 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.194612 23795 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.272940 21684 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:24.289749 21690 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:24.253738 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:24.257387 23811 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.267175 23618 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:24.302300 23735 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.335801 23618 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:24.341707 23735 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.344136 23735 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.336123 23834 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:24.348618 23834 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:24.351924 23691 consensus_meta.cc:287] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.535s user 0.010s sys 0.000s
I20240506 15:15:24.351975 23834 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:24.353221 23767 consensus_meta.cc:287] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.601s user 0.005s sys 0.005s
I20240506 15:15:24.353787 23767 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.357398 23549 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.369693 20780 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54786 (request call id 220) took 4794 ms. Trace:
I20240506 15:15:24.370180 20780 rpcz_store.cc:276] 0506 15:15:19.575553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:19.575730 (+ 177us) service_pool.cc:225] Handling call
0506 15:15:24.369578 (+4793848us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20240506 15:15:24.368244 23780 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.335135 23811 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.380365 23811 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.390705 23770 raft_consensus.cc:3032] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 LEADER]: Stepping down as leader of term 10
I20240506 15:15:24.391256 23770 raft_consensus.cc:729] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:24.392354 23770 consensus_queue.cc:260] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.335004 21688 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 3.
I20240506 15:15:24.394608 23584 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:24.395270 23583 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.395303 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.397634 23684 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:15:24.397722 23767 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.398097 23684 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:24.398514 23684 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 3 FOLLOWER]: Advancing to term 4
W20240506 15:15:24.400362 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 204964us
I20240506 15:15:24.403331 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:24.405371 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.406221 20880 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:24.402334 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.407886 20879 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:24.408648 23662 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:24.409045 23585 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.409272 23662 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.409550 23585 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.411393 23585 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.411644 23662 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:24.413965 21025 raft_consensus.cc:1226] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:24.417090 21027 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:24.419159 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47948 (request call id 1582) took 1141 ms. Trace:
I20240506 15:15:24.419770 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.419880 21029 rpcz_store.cc:276] 0506 15:15:23.278227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.279064 (+ 837us) service_pool.cc:225] Handling call
0506 15:15:24.418950 (+1139886us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b54001c2ad4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:24.419105 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1302960512}
W20240506 15:15:24.421183 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 195043us
I20240506 15:15:24.423305 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.424139 21030 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:24.426098 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:24.426954 20874 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.428941 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.430222 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.430757 21032 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:24.433149 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.438161 20922 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:24.438330 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.439728 20922 leader_election.cc:302] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:24.440487 21024 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:24.419090 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.452690 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.461175 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.461928 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.461941 21028 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:24.472730 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:24.474604 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 250159us
I20240506 15:15:24.475363 21024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47952 (request call id 1915) took 1109 ms. Trace:
I20240506 15:15:24.475729 21024 rpcz_store.cc:276] 0506 15:15:23.365712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.365836 (+ 124us) service_pool.cc:225] Handling call
0506 15:15:24.419088 (+1053252us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:24.448374 (+ 29286us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:24.475188 (+ 26814us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001c2ad0. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2eec59 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:24.475309 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2421547264}
I20240506 15:15:24.476490 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.476576 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.477967 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.477376 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:24.452867 23795 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:24.486191 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.460678 23699 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 15:15:24.488528 20957 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:24.482878 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:24.487098 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55214.2us
I20240506 15:15:24.483455 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.485285 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.488924 20874 raft_consensus.cc:2454] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:24.494987 20924 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.477340 23723 raft_consensus.cc:2726] T b540a7832fba49b5b2623616240c2711 P ecededcd854842d4867fda9fa63a4b66 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 15:15:24.517733 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.530736 20924 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.488716 23699 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:24.480638 23766 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240506 15:15:24.537622 20998 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.495822 23889 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:24.489548 23693 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:24.545578 20956 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:24.497539 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:24.541268 23786 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:24.554703 23783 raft_consensus.cc:1050] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:24.542074 23699 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:24.556246 23890 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:24.558053 20924 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.558851 23783 consensus_queue.cc:237] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.562139 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.562741 23861 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:24.553119 23786 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:24.564896 23684 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.578969 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.544378 23784 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:24.575907 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44874.5us
I20240506 15:15:24.546279 23676 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.578186 20998 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.546530 23751 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:24.586536 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.582794 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.583489 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.583806 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
W20240506 15:15:24.585544 23886 log.cc:924] Time spent T 75831e0251ec4bec9c8f3ca6c4a047f0 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20240506 15:15:24.587828 20874 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.588310 20950 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240506 15:15:24.603348 20924 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.578560 23786 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:24.607781 21690 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index 3 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:24.612037 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.612344 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.613559 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.614260 23755 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:24.615386 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.616830 20952 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:24.622270 20924 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.576227 23861 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.627398 23684 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:24.602087 23803 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.765s user 0.002s sys 0.009s
I20240506 15:15:24.628564 23803 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.630683 23803 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.623589 23676 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.636694 23751 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:24.590304 23585 raft_consensus.cc:2781] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:24.637130 23861 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:24.637614 23585 raft_consensus.cc:686] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:24.550359 23693 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.638540 23585 consensus_queue.cc:237] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.639686 23693 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:24.627580 23618 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.641742 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:24.641932 23618 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.643178 20954 raft_consensus.cc:1226] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240506 15:15:24.586844 23583 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:24.647862 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.646850 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.589260 23763 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:24.649699 20996 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:24.651067 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:24.651983 20875 raft_consensus.cc:3037] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:24.652643 23732 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.650754 23729 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.654209 23729 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.655903 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.587950 23872 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20240506 15:15:24.606142 23691 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:24.673493 21030 raft_consensus.cc:3037] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:24.667307 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.674255 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.670549 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.675467 20846 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.676220 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.674643 20956 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:24.651285 20845 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:24.679999 20876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b54b354e91941efb28b61b312923d41"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50474
I20240506 15:15:24.591323 20880 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 7: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:24.597060 20874 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:24.682687 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.680639 20876 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:24.683902 20876 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.689603 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.693492 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.697666 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.689626 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.705726 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:24.707684 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 56473.8us
I20240506 15:15:24.711045 23834 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:24.581194 23758 consensus_meta.cc:287] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.602s user 0.007s sys 0.004s
I20240506 15:15:24.712759 20875 raft_consensus.cc:2454] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 12.
I20240506 15:15:24.713421 21023 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:24.711930 23834 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:15:24.716871 21030 raft_consensus.cc:2454] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:24.648353 23583 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:24.714412 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.717656 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:24.718372 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:24.588464 23879 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:24.733088 20876 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.721956 20874 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:24.736488 20846 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.728950 23583 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.739560 23618 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:24.744675 23618 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:24.745055 23618 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.681294 23813 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:24.652689 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:24.656173 23729 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.740886 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:24.750241 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.739938 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.750111 20876 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.716910 23662 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.751448 21030 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.754658 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.773519 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.688979 23691 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:24.781502 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 75969.4us
I20240506 15:15:24.771200 21030 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:24.787820 20845 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.782663 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.784854 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.751477 23811 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.758252 23751 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:24.796906 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:24.797114 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47952 (request call id 1935) took 1183 ms. Trace:
I20240506 15:15:24.798278 21023 rpcz_store.cc:276] 0506 15:15:23.613786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.618173 (+ 4387us) service_pool.cc:225] Handling call
0506 15:15:24.796956 (+1178783us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540039d0d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:24.797058 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2366108416}
I20240506 15:15:24.801115 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.801616 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.802338 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.803696 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.807610 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:24.793212 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 55716.4us
I20240506 15:15:24.809265 20998 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.746102 23693 consensus_queue.cc:1046] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20240506 15:15:24.740600 23763 consensus_queue.cc:1046] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20240506 15:15:24.811244 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.747749 23690 consensus_queue.cc:1046] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20240506 15:15:24.818053 20953 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 10: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:24.818253 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.818825 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.819272 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.722112 23758 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.820034 20953 raft_consensus.cc:1226] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:24.821403 20996 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.821738 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:24.823381 20998 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:24.788128 23846 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:24.792500 23834 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:24.753558 23813 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.825610 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:24.826185 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 43614.7us
I20240506 15:15:24.829921 20846 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.830767 20958 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 12: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:24.834029 23811 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 15:15:24.833227 20996 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.825421 23784 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:24.837543 23584 raft_consensus.cc:2781] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Leader election won for term 12
I20240506 15:15:24.832131 23732 raft_consensus.cc:1050] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:24.831091 23834 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:24.869410 23872 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:24.826965 23743 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.826506 23679 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 15:15:24.897015 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 63900.8us
I20240506 15:15:24.869154 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:24.893580 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 68033.9us
I20240506 15:15:24.895514 23691 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.895747 23695 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:24.916007 23695 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:24.916563 23695 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:24.904583 20921 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:24.923537 20921 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:15:24.870160 23902 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20240506 15:15:24.924800 23890 raft_consensus.cc:2781] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:24.923995 23695 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.908004 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:24.925822 23879 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:24.876344 20776 catalog_manager.cc:5484] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 5 to 7, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 7 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:24.930406 20921 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:24.910354 21023 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 11. Current term is 12. Ops: []
I20240506 15:15:24.900499 23872 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:24.925379 23890 raft_consensus.cc:686] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [term 10 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:24.912554 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.931236 20776 catalog_manager.cc:5484] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 7 to 9, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129). New cstate: current_term: 9 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:24.870798 23763 raft_consensus.cc:3032] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:24.952389 23763 raft_consensus.cc:729] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:24.953292 23763 consensus_queue.cc:260] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.954497 23763 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:24.948324 23699 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.958338 23699 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:24.959969 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 91099.5us
I20240506 15:15:24.959992 23679 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:24.960636 21688 raft_consensus.cc:1226] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:24.949399 23873 raft_consensus.cc:3032] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:24.961843 23873 raft_consensus.cc:729] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:24.962760 23872 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.963961 23872 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:24.963893 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:24.961283 20846 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.969894 23837 raft_consensus.cc:2726] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:24.970325 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.972359 23783 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:24.976344 21023 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:24.977699 23695 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.973711 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.908674 20996 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.979344 20846 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:24.980458 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 71883.8us
I20240506 15:15:24.981175 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:24.983067 20996 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:24.983855 21656 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:24.938256 20776 catalog_manager.cc:5484] T 23b7301301694836b2d08bf22e3f2e38 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "00fe71c1675441379f5f2c7490f9e63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240506 15:15:24.989367 23837 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:24.990288 23699 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240506 15:15:24.990828 23783 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:24.941454 23834 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.942379 23732 consensus_queue.cc:237] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:24.945848 23693 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:24.996340 23693 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.879055 23583 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.998450 23693 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.946329 23770 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:24.999522 23583 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:24.946646 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:24.947048 23729 consensus_queue.cc:1046] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240506 15:15:24.962760 23873 consensus_queue.cc:260] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.948220 23813 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.906880 23766 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:24.879350 23758 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:24.948884 23784 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.001538 23873 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:25.005129 23784 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.008435 23837 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:24.902037 23786 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:24.949398 23755 raft_consensus.cc:2741] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240506 15:15:24.951093 23890 consensus_queue.cc:237] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.011785 23786 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:25.014390 20950 raft_consensus.cc:2454] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 11.
I20240506 15:15:25.015698 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.016605 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
W20240506 15:15:25.029070 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 114973us
I20240506 15:15:25.031131 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.033484 21028 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 10: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:25.034471 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.035100 20881 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:25.035105 20874 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:25.035367 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.036389 20879 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.038772 20846 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.031802 20776 catalog_manager.cc:5484] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:25.040766 21654 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.050180 20879 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:25.028664 23695 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.050701 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.055851 21024 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:25.034514 23859 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240506 15:15:25.058656 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.051487 23811 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:25.036988 23783 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.040372 23777 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:25.064383 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.039062 23584 raft_consensus.cc:686] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [term 12 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.080197 23584 consensus_queue.cc:237] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.088531 21027 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:25.089651 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:25.089681 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:25.091185 21029 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:25.051606 23858 raft_consensus.cc:2781] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader election won for term 11
I20240506 15:15:25.078624 23846 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.043546 23755 consensus_queue.cc:1046] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20240506 15:15:25.062043 23735 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240506 15:15:25.106729 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 108753us
I20240506 15:15:25.071911 23679 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.109189 20996 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.115105 20998 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:25.121541 21026 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:25.099365 23811 raft_consensus.cc:686] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:25.129815 20879 raft_consensus.cc:1226] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 3. Current term is 4. Ops: []
I20240506 15:15:25.056825 23618 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.107811 23858 raft_consensus.cc:686] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:25.149979 23859 consensus_queue.cc:1046] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:25.151055 23770 consensus_queue.cc:1046] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240506 15:15:25.152088 23770 raft_consensus.cc:3032] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:25.152475 23770 raft_consensus.cc:729] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:25.139572 23662 raft_consensus.cc:2781] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:25.153402 23770 consensus_queue.cc:260] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.155230 20955 raft_consensus.cc:2379] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 8: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:25.155757 23618 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:25.138898 23585 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:25.169095 20949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "286345f896bc43ec975183896aee147c"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:25.169991 20949 raft_consensus.cc:595] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.170418 20949 raft_consensus.cc:603] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:25.115140 23729 consensus_queue.cc:1046] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20240506 15:15:25.119962 23689 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240506 15:15:25.174755 23618 consensus_queue.cc:1046] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240506 15:15:25.181567 23618 raft_consensus.cc:3032] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:25.181839 23859 raft_consensus.cc:3032] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Stepping down as leader of term 3
I20240506 15:15:25.181982 23618 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:25.182350 23859 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.182919 23618 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.183187 23859 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:25.174672 23770 raft_consensus.cc:3037] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:25.135310 23583 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20240506 15:15:25.190187 20881 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:25.165081 23585 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:25.197829 23763 consensus_queue.cc:1046] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.011s
I20240506 15:15:25.209442 23763 consensus_queue.cc:1225] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer 7610624a16b44896b6aacea1928ab20e log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240506 15:15:25.210122 23763 raft_consensus.cc:3032] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:25.210552 23763 raft_consensus.cc:729] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:25.202296 23811 consensus_queue.cc:237] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.211663 23763 consensus_queue.cc:260] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.228021 23763 raft_consensus.cc:3037] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.202490 23662 raft_consensus.cc:686] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.186722 23618 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.244356 21032 raft_consensus.cc:1226] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 11. Current term is 12. Ops: []
I20240506 15:15:25.186656 23859 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 3 FOLLOWER]: Advancing to term 4
I20240506 15:15:25.165781 23861 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.242962 21023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:25.243485 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:25.184530 23858 consensus_queue.cc:237] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:25.253834 21023 raft_consensus.cc:595] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.264441 23884 raft_consensus.cc:3032] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 LEADER]: Stepping down as leader of term 11
I20240506 15:15:25.264953 23884 raft_consensus.cc:729] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:25.244204 23901 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:25.255331 20998 leader_election.cc:302] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
W20240506 15:15:25.255837 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 106883us
I20240506 15:15:25.267742 21030 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:25.200346 23585 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:25.266253 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 202004us
I20240506 15:15:25.270658 20846 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.271809 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.276392 20846 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.276312 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:25.277352 20950 raft_consensus.cc:3037] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.281540 20949 raft_consensus.cc:1261] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:25.283205 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.275903 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:25.295307 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.298323 20846 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:25.267139 23766 raft_consensus.cc:3032] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Stepping down as leader of term 4
I20240506 15:15:25.350652 21690 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:25.305718 20924 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:25.358062 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 52456.3us
I20240506 15:15:25.361388 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:25.302260 20950 raft_consensus.cc:2454] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:25.370656 20949 raft_consensus.cc:1226] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:25.370431 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:25.375460 20949 raft_consensus.cc:1261] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:25.373937 20950 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:25.378726 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:25.284134 23846 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.294387 23679 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:25.291122 23729 raft_consensus.cc:2781] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:25.349530 23735 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.291373 23684 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.351853 23691 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.293875 23834 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:25.265157 23662 consensus_queue.cc:237] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.352825 23766 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.367201 23693 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:25.374253 23813 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:25.293481 23777 consensus_queue.cc:1046] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.114s
I20240506 15:15:25.464370 20949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81702cae5854497d947832321f954762"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:25.517941 20949 raft_consensus.cc:595] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.518365 20949 raft_consensus.cc:603] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:25.282297 23901 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:25.478674 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 112069us
I20240506 15:15:25.533077 23766 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.536394 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.537139 20879 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:25.539726 21031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:25.540331 21031 raft_consensus.cc:595] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.537178 20881 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:25.480408 23803 raft_consensus.cc:3032] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:25.547564 20879 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:25.554898 20998 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:25.510720 23729 raft_consensus.cc:686] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [term 9 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.581146 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.551270 23803 raft_consensus.cc:729] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.587308 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.565548 23584 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.004s
I20240506 15:15:25.591555 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.593114 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.402853 23884 consensus_queue.cc:260] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.593391 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.529920 23872 consensus_queue.cc:1046] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20240506 15:15:25.599671 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.598505 23583 raft_consensus.cc:979] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:25.585032 20881 raft_consensus.cc:2932] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index 3 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:25.520148 23693 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:25.604054 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.626188 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.634332 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.635191 20875 raft_consensus.cc:2379] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 10: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:25.594239 23743 raft_consensus.cc:2781] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:25.636593 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.637087 20882 raft_consensus.cc:1226] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:25.648102 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.651018 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.590232 23861 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:25.652273 20875 raft_consensus.cc:1226] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:25.653142 20924 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.658448 20875 raft_consensus.cc:1226] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:25.655336 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.649688 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.680361 20880 raft_consensus.cc:2379] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 12: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:25.664093 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.665161 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:25.663988 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:25.669968 20924 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:25.669745 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 204817us
I20240506 15:15:25.641456 23872 raft_consensus.cc:3032] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:25.684427 20924 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.660023 23767 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.686611 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.654515 23926 raft_consensus.cc:1067] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:25.676616 23743 raft_consensus.cc:686] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:25.689805 20877 raft_consensus.cc:1226] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:25.690508 20882 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:25.690986 20880 raft_consensus.cc:1226] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240506 15:15:25.691687 20883 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:25.687614 23873 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.685539 23872 raft_consensus.cc:729] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:25.718019 20878 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 4. (term mismatch)
I20240506 15:15:25.718336 20878 pending_rounds.cc:76] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 3
I20240506 15:15:25.718509 20878 pending_rounds.cc:97] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240506 15:15:25.718706 20878 raft_consensus.cc:2866] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:25.731036 20878 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240506 15:15:25.733407 20846 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.739567 20875 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240506 15:15:25.673077 23729 consensus_queue.cc:237] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.678040 23803 consensus_queue.cc:260] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:25.764720 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 78222.1us
I20240506 15:15:25.717234 23813 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:25.740304 23735 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:25.738296 23873 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20240506 15:15:25.762894 23763 consensus_meta.cc:287] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.534s user 0.006s sys 0.006s
I20240506 15:15:25.780889 21028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
W20240506 15:15:25.772346 23925 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
W20240506 15:15:25.626334 23877 consensus_queue.cc:1159] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7610624a16b44896b6aacea1928ab20e" responder_term: 12 status { last_received { term: 11 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:15:25.771765 23872 consensus_queue.cc:260] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:25.785538 23924 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240506 15:15:25.785655 23872 raft_consensus.cc:3037] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:25.786878 23766 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:15:25.789189 23920 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20240506 15:15:25.782546 21028 raft_consensus.cc:595] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:25.792773 23693 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:25.765452 23901 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:25.736699 23783 consensus_queue.cc:1046] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240506 15:15:25.793995 23884 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:25.794225 23783 raft_consensus.cc:3032] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:25.794700 23783 raft_consensus.cc:729] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:25.776376 23735 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:25.807976 23618 consensus_meta.cc:287] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.556s user 0.008s sys 0.002s
I20240506 15:15:25.795519 23783 consensus_queue.cc:260] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.807976 23834 raft_consensus.cc:3032] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:25.812322 23834 raft_consensus.cc:729] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:25.813395 23834 consensus_queue.cc:260] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.801478 23691 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.801307 23735 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:25.778754 23743 consensus_queue.cc:237] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:25.824448 23770 consensus_meta.cc:287] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.632s user 0.012s sys 0.000s
W20240506 15:15:25.835165 23859 consensus_meta.cc:287] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.581s user 0.009s sys 0.000s
W20240506 15:15:25.842257 23922 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.267s user 0.001s sys 0.001s
W20240506 15:15:25.840132 23923 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.270s user 0.000s sys 0.004s
I20240506 15:15:25.766716 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.914227 23770 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.903824 20874 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:25.910243 23926 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:25.911376 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:25.917312 21023 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 9.
I20240506 15:15:25.917049 23926 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.927510 23926 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:25.933036 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 166221us
I20240506 15:15:25.934774 20924 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:25.936882 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:25.953619 21031 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:25.956066 21032 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:25.957777 21653 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:25.958653 23926 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:25.959693 23684 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:25.960016 23857 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:25.959693 23926 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.968401 23877 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:25.960625 23857 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:25.974937 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:25.971289 23857 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:25.995855 23783 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:25.960695 23684 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:25.997408 20878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:26.001508 20883 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:26.002177 20878 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:26.002759 20878 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.014091 23684 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.020329 23926 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:26.022146 23926 consensus_queue.cc:1035] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20240506 15:15:26.022527 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 291061us
W20240506 15:15:26.027159 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 92643us
I20240506 15:15:26.027743 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:26.030599 23890 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:26.035044 21690 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240506 15:15:26.036765 23890 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.040052 23890 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.040113 20955 raft_consensus.cc:2454] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:26.051285 23859 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:26.053208 23729 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:26.057247 23936 raft_consensus.cc:979] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:26.058171 23662 raft_consensus.cc:1067] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:26.058930 23834 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:26.062114 23803 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.063102 23662 raft_consensus.cc:1050] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: attempting to promote NON_VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e to VOTER
I20240506 15:15:26.064435 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.065172 21031 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:26.066507 23662 consensus_queue.cc:237] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:26.083076 21031 raft_consensus.cc:1261] T 10845b6e0cc149f5a5e40605a2ea7e49 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 10 index: 4. (index mismatch)
I20240506 15:15:26.087293 20950 raft_consensus.cc:1261] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240506 15:15:26.087602 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.085747 20952 raft_consensus.cc:1261] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:26.090443 23890 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:26.088316 21032 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:26.084376 20958 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:26.096689 20878 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.091403 23890 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } }
I20240506 15:15:26.101296 20875 raft_consensus.cc:3037] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:26.105964 20876 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:26.107913 20878 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.045928 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:26.120815 20779 catalog_manager.cc:5484] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240506 15:15:26.116958 23585 consensus_meta.cc:287] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.822s user 0.006s sys 0.014s
I20240506 15:15:26.126827 20949 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:26.129709 23735 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.132856 23735 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.135669 23585 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.145124 21024 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:26.146013 20952 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 4. Current term is 5. Ops: []
I20240506 15:15:26.148965 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.141112 23857 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:26.126935 23584 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20240506 15:15:26.161548 23890 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.164044 23813 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.171262 23813 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.173663 23813 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.164764 23952 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.175029 23952 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.176311 21688 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 10: Already voted for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in this term.
I20240506 15:15:26.177368 23952 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.113046 23926 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20240506 15:15:26.178498 23857 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:26.166422 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 44020.1us
I20240506 15:15:26.170112 23585 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:26.182740 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:26.183389 21023 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:26.158999 21024 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:26.186245 20875 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240506 15:15:26.200536 23890 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.201246 23890 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.204047 23890 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.213213 21023 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
W20240506 15:15:26.218307 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 484833us
I20240506 15:15:26.223171 23735 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:26.223651 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.223920 23837 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:26.214169 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.218935 20779 catalog_manager.cc:5484] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:26.228133 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:26.233353 21653 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:26.234681 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.235612 20958 raft_consensus.cc:2379] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 8: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:26.239362 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.240208 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.247573 20881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:26.248463 20881 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:26.249032 20881 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.240160 20878 raft_consensus.cc:2354] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:26.254161 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.254985 20958 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:26.257272 23699 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:26.257501 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:26.257849 23699 raft_consensus.cc:686] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:26.258385 20880 raft_consensus.cc:2354] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8.
I20240506 15:15:26.258880 23699 consensus_queue.cc:237] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:26.262184 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.263041 20955 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:26.264305 20953 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:26.264401 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.264998 20949 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240506 15:15:26.265334 20956 raft_consensus.cc:2354] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8.
I20240506 15:15:26.265398 20949 pending_rounds.cc:76] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Aborting all ops after (but not including) 4
I20240506 15:15:26.266580 20952 raft_consensus.cc:2454] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:26.265846 20949 pending_rounds.cc:97] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240506 15:15:26.274643 20949 raft_consensus.cc:2866] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:26.265885 23735 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:26.278478 23952 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.284801 23837 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240506 15:15:26.291565 23952 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:26.292143 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 64211.9us
I20240506 15:15:26.293170 20957 raft_consensus.cc:1261] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240506 15:15:26.293888 23952 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.294759 23786 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:26.308665 20776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54780 (request call id 226) took 2991 ms. Trace:
I20240506 15:15:26.290830 23735 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:26.314121 23735 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:26.323429 20958 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
I20240506 15:15:26.279925 20881 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.323856 23884 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240506 15:15:26.329599 23786 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:26.333567 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:26.309206 20776 rpcz_store.cc:276] 0506 15:15:23.317231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:23.335557 (+ 18326us) service_pool.cc:225] Handling call
0506 15:15:26.308375 (+2972818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:25.260747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:25.261225 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:25.261255 (+ 30us) write_op.cc:435] Acquired schema lock
0506 15:15:25.261271 (+ 16us) tablet.cc:660] Decoding operations
0506 15:15:25.261842 (+ 571us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:25.261909 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:25.261934 (+ 25us) tablet.cc:683] Acquiring locks for 4 operations
0506 15:15:25.262288 (+ 354us) tablet.cc:699] Row locks acquired
0506 15:15:25.262310 (+ 22us) write_op.cc:260] PREPARE: finished
0506 15:15:25.262445 (+ 135us) write_op.cc:270] Start()
0506 15:15:25.262558 (+ 113us) write_op.cc:276] Timestamp: P: 1715008525262401 usec, L: 0
0506 15:15:25.262589 (+ 31us) op_driver.cc:331] REPLICATION: starting
0506 15:15:25.264887 (+ 2298us) log.cc:842] Serialized 1702 byte log entry
0506 15:15:25.841542 (+576655us) op_driver.cc:444] REPLICATION: finished
0506 15:15:25.929816 (+ 88274us) write_op.cc:301] APPLY: starting
0506 15:15:25.930091 (+ 275us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:25.930435 (+ 344us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:25.930661 (+ 226us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:25.931818 (+ 1157us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:25.964562 (+ 32744us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0506 15:15:25.967810 (+ 3248us) write_op.cc:312] APPLY: finished
0506 15:15:26.024457 (+ 56647us) log.cc:842] Serialized 41 byte log entry
0506 15:15:26.024846 (+ 389us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:26.025161 (+ 315us) write_op.cc:454] Released schema lock
0506 15:15:26.025267 (+ 106us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:26.025950 (+ 683us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:26.026004 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":132188,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":87707,"apply.run_cpu_time_us":5672,"apply.run_wall_time_us":96779,"num_ops":4,"prepare.queue_time_us":160399,"prepare.run_cpu_time_us":87249,"prepare.run_wall_time_us":305548,"raft.queue_time_us":49342,"raft.run_cpu_time_us":8328,"raft.run_wall_time_us":88205,"replication_time_us":578809,"thread_start_us":572300,"threads_started":4,"wal-append.queue_time_us":395407}]]}
I20240506 15:15:26.344902 20952 raft_consensus.cc:1261] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240506 15:15:26.336306 23926 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.360205 20881 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.366943 23735 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.370201 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:26.355298 23952 raft_consensus.cc:2741] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240506 15:15:26.371136 21023 raft_consensus.cc:3032] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:26.371592 21023 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:26.373139 20998 leader_election.cc:398] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:26.373504 20998 leader_election.cc:401] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:26.374397 23729 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 9
I20240506 15:15:26.374625 20998 leader_election.cc:398] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8.
I20240506 15:15:26.357545 23857 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20240506 15:15:26.375001 20998 leader_election.cc:401] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240506 15:15:26.376977 20878 raft_consensus.cc:1261] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:26.377147 23937 raft_consensus.cc:2726] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8.
I20240506 15:15:26.378973 23931 consensus_queue.cc:1035] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240506 15:15:26.379499 21023 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.380661 21023 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:26.381901 23585 consensus_queue.cc:1159] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "00fe71c1675441379f5f2c7490f9e63a" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:15:26.389153 23729 raft_consensus.cc:2726] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:26.389750 20874 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:26.394044 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:26.394739 21026 raft_consensus.cc:3032] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:26.395066 21026 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:26.395884 21026 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.405997 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:26.412138 21026 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.414809 20921 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:26.359093 23926 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.421451 21023 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:26.433826 23585 consensus_queue.cc:1035] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:26.440551 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.445202 20845 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.422369 23770 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:26.453318 20955 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:26.458933 20883 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:26.461190 20878 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:26.459527 20876 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:26.465539 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.457144 23770 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:26.449144 23846 consensus_queue.cc:1035] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20240506 15:15:26.464879 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.475972 20881 raft_consensus.cc:1261] T 1a88ec9e65b24b64be49384639f4f9f9 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:26.488718 21025 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:26.463061 23584 consensus_queue.cc:1035] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20240506 15:15:26.498057 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.477222 23735 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.497270 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.508679 20924 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:26.507939 23770 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.517347 23811 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:26.517939 23811 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:26.518987 23811 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.520437 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.521934 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:26.522560 21028 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
W20240506 15:15:26.523512 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 44187.1us
I20240506 15:15:26.524385 21685 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 LEARNER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:26.525205 23926 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:26.526276 20921 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:26.516614 23955 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20240506 15:15:26.531798 21023 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:26.527146 23952 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:26.544523 23743 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:26.571321 23784 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:26.582664 23662 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240506 15:15:26.595909 23961 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.084s user 0.000s sys 0.002s
I20240506 15:15:26.539907 23890 raft_consensus.cc:2726] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:26.621227 21028 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:26.539400 23952 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:26.629695 20949 raft_consensus.cc:1261] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240506 15:15:26.596326 23786 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
W20240506 15:15:26.597748 23962 log.cc:924] Time spent T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20240506 15:15:26.627780 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:26.522279 23937 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
W20240506 15:15:26.600746 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 110886us
I20240506 15:15:26.647367 21024 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:26.653518 21026 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:26.652590 23846 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.659531 23952 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:26.676527 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 94511.4us
I20240506 15:15:26.689193 20921 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:26.691802 20845 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.697151 20845 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.702153 20845 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.692771 23834 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.704535 23928 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:26.701426 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:26.693579 20924 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.889s (SENT) [suppressed 2 similar messages]
I20240506 15:15:26.695494 23770 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:26.709661 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.703821 23834 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.714692 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.674535 23846 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.716395 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.718531 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.707742 23811 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:26.707091 23735 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:26.707990 23926 raft_consensus.cc:2741] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20240506 15:15:26.716634 23765 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.675630 23786 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.711745 23770 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:26.732787 23765 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.724808 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.710718 23937 consensus_queue.cc:1035] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240506 15:15:26.734988 23811 raft_consensus.cc:686] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:26.739984 23846 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.739653 23811 consensus_queue.cc:237] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.736387 23770 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:26.742141 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:26.743077 20879 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
W20240506 15:15:26.754004 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 46274.8us
I20240506 15:15:26.755178 20996 leader_election.cc:398] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8.
I20240506 15:15:26.761636 23786 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:26.762856 23970 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.240s user 0.001s sys 0.000s
W20240506 15:15:26.764740 23964 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.251s user 0.000s sys 0.002s
W20240506 15:15:26.765255 23968 log.cc:924] Time spent T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20240506 15:15:26.771744 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.737938 23735 raft_consensus.cc:686] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:26.776743 23735 consensus_queue.cc:237] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.790896 23834 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.793187 23809 consensus_queue.cc:1035] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20240506 15:15:26.796402 23966 log.cc:924] Time spent T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.278s user 0.002s sys 0.000s
I20240506 15:15:26.798269 23662 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:26.806586 23809 consensus_queue.cc:1035] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240506 15:15:26.779078 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 77916.6us
W20240506 15:15:26.810694 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 117139us
I20240506 15:15:26.817951 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.822366 20950 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.756788 23928 raft_consensus.cc:686] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:26.824172 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.824923 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.829330 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.824208 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.830209 20958 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:26.830575 20951 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:26.834575 20949 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:26.812981 23765 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:26.853417 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.854310 20952 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:26.856364 21023 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:26.865047 20951 raft_consensus.cc:1261] T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:26.881668 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:26.897316 23985 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.897964 23985 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:26.870194 23926 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.900023 23985 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:26.908948 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.915529 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:26.903012 23735 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:26.926635 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:26.927397 21688 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
I20240506 15:15:26.947105 20958 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:26.947244 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:26.914299 23662 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:26.941192 23984 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:26.920203 23735 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:26.955489 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.965678 20954 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:26.955394 23890 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:26.962492 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.959023 20950 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:26.956722 23743 raft_consensus.cc:979] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:26.975996 20958 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:26.978843 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884787c4c4b474685c7e903e8789e90" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:26.919608 23926 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:26.919895 23770 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.995878 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:26.969641 23735 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:26.901818 23928 consensus_queue.cc:237] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.004338 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:27.005360 20950 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:26.975831 23684 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:26.981552 23884 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:27.024771 21030 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 4.
W20240506 15:15:27.025450 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 54450us
I20240506 15:15:26.975587 23662 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.011610 23952 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.020992 23684 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:27.088706 23952 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:27.088747 23987 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20240506 15:15:27.097671 20950 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
W20240506 15:15:27.100659 23991 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20240506 15:15:27.107236 20954 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:26.997740 23890 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:26.968166 23984 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.015944 23803 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20240506 15:15:27.119026 21023 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240506 15:15:27.057442 23884 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.137115 23884 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.137881 23858 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240506 15:15:27.110548 23684 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Advancing to term 5
W20240506 15:15:27.159780 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 132475us
I20240506 15:15:27.183223 20954 raft_consensus.cc:1261] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20240506 15:15:27.183856 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 227151us
I20240506 15:15:27.201855 23803 consensus_queue.cc:1035] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:27.200786 23984 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.201637 23584 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:27.201871 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:27.202992 23890 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.203859 23770 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.215301 23743 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:27.215974 23743 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:27.216584 23743 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:27.218587 23803 consensus_queue.cc:1035] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:27.220444 23986 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:27.221050 23986 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.223416 23986 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:27.257915 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 279398us
I20240506 15:15:27.293648 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:27.297784 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:27.298530 21028 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:27.299924 23926 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:27.314584 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:27.315311 21032 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:27.316505 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:27.304464 23662 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:27.327229 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:27.328114 21685 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 4.
I20240506 15:15:27.328389 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:27.294318 21688 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:27.329633 20996 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:27.330791 23803 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 15:15:27.311385 23735 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.331354 23803 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:27.331774 23803 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:27.290805 23952 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:27.332623 23952 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.310016 23684 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:27.334645 23952 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.335613 23684 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:27.335698 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:27.336619 20953 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:27.342041 23837 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:27.342671 23837 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.343611 23786 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.344202 23786 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.345005 20924 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:27.345782 23770 consensus_queue.cc:1046] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 15:15:27.346117 23985 raft_consensus.cc:482] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.346582 24000 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:27.346769 23985 raft_consensus.cc:504] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.346233 23786 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.336014 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 86) took 3271 ms. Trace:
I20240506 15:15:27.350983 21028 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:27.315840 23993 raft_consensus.cc:1067] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:27.365934 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:27.363813 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:27.366477 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:27.366714 20953 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 12.
I20240506 15:15:27.367396 20882 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:27.372658 23984 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:27.373518 23984 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.377673 20924 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:27.379540 20924 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:27.381107 23765 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:27.383319 23890 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240506 15:15:27.385161 20924 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:27.386278 23890 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:27.376111 23984 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.388906 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:27.390002 23984 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:15:27.394333 23992 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20240506 15:15:27.395089 23985 leader_election.cc:288] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.395296 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:27.395964 21685 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Advancing to term 5
I20240506 15:15:27.360580 23786 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:27.396555 23743 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.399298 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:27.398169 20996 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:27.400918 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:27.403010 20876 raft_consensus.cc:2379] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:27.404560 21028 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:27.361323 23834 raft_consensus.cc:482] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.410446 20882 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 11.
I20240506 15:15:27.417025 21032 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:27.405687 23786 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.419306 23743 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:27.420471 23786 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.413761 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:27.421443 23735 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.421542 21031 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 12.
I20240506 15:15:27.422837 23834 raft_consensus.cc:504] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.424902 23834 leader_election.cc:288] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.353526 20779 rpcz_store.cc:276] 0506 15:15:24.064350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:24.064712 (+ 362us) service_pool.cc:225] Handling call
0506 15:15:27.335956 (+3271244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:26.514326 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:26.515097 (+ 771us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:26.515159 (+ 62us) write_op.cc:435] Acquired schema lock
0506 15:15:26.515174 (+ 15us) tablet.cc:660] Decoding operations
0506 15:15:26.515719 (+ 545us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:26.518147 (+ 2428us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:26.518186 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0506 15:15:26.518844 (+ 658us) tablet.cc:699] Row locks acquired
0506 15:15:26.518859 (+ 15us) write_op.cc:260] PREPARE: finished
0506 15:15:26.519019 (+ 160us) write_op.cc:270] Start()
0506 15:15:26.519147 (+ 128us) write_op.cc:276] Timestamp: P: 1715008526518954 usec, L: 0
0506 15:15:26.519166 (+ 19us) op_driver.cc:331] REPLICATION: starting
0506 15:15:26.519851 (+ 685us) log.cc:842] Serialized 915 byte log entry
0506 15:15:26.698896 (+179045us) op_driver.cc:444] REPLICATION: finished
0506 15:15:26.833519 (+134623us) write_op.cc:301] APPLY: starting
0506 15:15:26.833598 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:26.833711 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:26.833842 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:26.838027 (+ 4185us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:26.840219 (+ 2192us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 15:15:26.840296 (+ 77us) write_op.cc:312] APPLY: finished
0506 15:15:26.840493 (+ 197us) log.cc:842] Serialized 29 byte log entry
0506 15:15:26.840650 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:26.840740 (+ 90us) write_op.cc:454] Released schema lock
0506 15:15:26.840780 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:26.841089 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:26.841122 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":383885,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":125900,"apply.run_cpu_time_us":2237,"apply.run_wall_time_us":7892,"num_ops":2,"prepare.queue_time_us":248586,"prepare.run_cpu_time_us":3368,"prepare.run_wall_time_us":7979,"replication_time_us":175915,"thread_start_us":221054,"threads_started":2}]]}
I20240506 15:15:27.427795 23837 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:27.460577 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 51635us
W20240506 15:15:27.482733 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60763.4us
I20240506 15:15:27.443560 23662 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240506 15:15:27.501071 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 86457.7us
I20240506 15:15:27.492137 23783 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:27.484440 23834 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.491082 23662 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:27.572737 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 60283.1us
I20240506 15:15:27.489429 23765 consensus_queue.cc:1035] T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
W20240506 15:15:27.586441 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 83844.9us
I20240506 15:15:27.516371 23662 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:27.604420 20877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:27.606593 20921 leader_election.cc:302] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:27.626701 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:27.511755 23783 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:27.665625 24013 log.cc:924] Time spent T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240506 15:15:27.586539 23928 raft_consensus.cc:482] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.669754 23928 raft_consensus.cc:504] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.612329 23993 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:27.623607 23813 raft_consensus.cc:2781] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:27.679435 20877 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:27.581807 23834 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.680212 20877 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
W20240506 15:15:27.682778 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 89180.2us
I20240506 15:15:27.679596 23928 leader_election.cc:288] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.667879 23803 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:27.908003 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 297566us
I20240506 15:15:27.914466 20921 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:27.925158 20921 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:27.918820 21688 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:27.946794 21023 raft_consensus.cc:2379] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
W20240506 15:15:27.951651 24012 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20240506 15:15:27.944571 23969 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240506 15:15:27.952656 24029 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:27.953716 24015 log.cc:924] Time spent T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.368s user 0.001s sys 0.000s
I20240506 15:15:27.953399 24029 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.955621 24029 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:27.956243 23834 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:27.959388 23783 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:27.952937 24025 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20240506 15:15:27.909950 24016 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:27.960942 24016 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:27.963665 24016 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:27.965409 23803 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:27.681631 23993 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.984777 20877 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:27.951306 23813 raft_consensus.cc:686] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:28.024093 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:28.032418 24026 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.033111 24026 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:28.024281 24022 log.cc:924] Time spent T a9f4d07ea30646d0805b63bb641358be P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20240506 15:15:28.035292 24026 leader_election.cc:288] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.036363 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:28.037457 23662 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.038093 24028 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.363s
W20240506 15:15:28.040033 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 245257us
I20240506 15:15:28.040521 23662 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.049176 24030 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.050930 23993 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.053622 24000 raft_consensus.cc:3032] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:28.054175 24000 raft_consensus.cc:729] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:28.061350 24000 consensus_queue.cc:260] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:28.043462 24019 log.cc:924] Time spent T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.355s user 0.000s sys 0.000s
I20240506 15:15:27.977644 23846 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:28.064152 23846 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.082523 23884 raft_consensus.cc:2781] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:28.083417 23884 raft_consensus.cc:686] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:28.084007 23846 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.095023 23770 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240506 15:15:28.095679 23770 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:28.097514 24000 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:28.115465 24023 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.122327 23884 consensus_queue.cc:237] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.096140 23770 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Advancing to term 13
W20240506 15:15:28.139957 24027 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.220s user 0.003s sys 0.000s
I20240506 15:15:28.159816 20779 catalog_manager.cc:5484] T 81702cae5854497d947832321f954762 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 10 to 11, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 11 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240506 15:15:28.195156 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 270115us
W20240506 15:15:28.179958 20876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.009s sys 0.014s
W20240506 15:15:28.201061 20876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.009s sys 0.014s
I20240506 15:15:28.203588 20996 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:28.206094 23662 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240506 15:15:28.206516 23662 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:28.206846 23662 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Advancing to term 13
I20240506 15:15:28.210083 23999 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.210803 23999 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:28.213194 21685 consensus_meta.cc:287] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.782s user 0.011s sys 0.001s
I20240506 15:15:27.994935 23813 consensus_queue.cc:237] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.256072 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:28.256887 20950 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:28.211738 23770 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.278746 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:28.279757 20954 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:28.185086 20779 catalog_manager.cc:5484] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 6 to 7, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131), now has a pending config: VOTER 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129), VOTER ecededcd854842d4867fda9fa63a4b66 (127.19.67.130), VOTER 7610624a16b44896b6aacea1928ab20e (127.19.67.131), VOTER 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: NON_VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.288693 20779 catalog_manager.cc:5484] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: config changed from index 3 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.296201 20779 catalog_manager.cc:5484] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e reported cstate change: term changed from 8 to 10, leader changed from 7610624a16b44896b6aacea1928ab20e (127.19.67.131) to 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132). New cstate: current_term: 10 leader_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240506 15:15:28.213774 21685 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:28.308504 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.309376 20952 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 8: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:28.296790 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.316318 20949 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 10: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:28.273828 21684 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.335309 20877 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.337301 23585 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.337826 23585 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.334519 20954 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:28.349901 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:28.359659 23585 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:28.370721 20845 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.010s sys 0.012s
W20240506 15:15:28.371146 20845 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.010s sys 0.012s
I20240506 15:15:28.376076 20950 raft_consensus.cc:2454] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
W20240506 15:15:28.394099 20924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.005s sys 0.018s
W20240506 15:15:28.413443 20924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.006s sys 0.018s
W20240506 15:15:28.421994 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 708861us
I20240506 15:15:28.423059 20924 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:28.407836 24001 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.040s sys 0.050s
I20240506 15:15:28.430622 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.447582 20924 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:28.444885 24001 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s user 0.041s sys 0.050s
I20240506 15:15:28.454757 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:28.454113 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 445910us
W20240506 15:15:28.468824 20997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.014s sys 0.023s
W20240506 15:15:28.469347 20997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.014s sys 0.023s
I20240506 15:15:28.443337 23834 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.477792 20924 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
W20240506 15:15:28.505568 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 54151.1us
I20240506 15:15:28.508208 20996 leader_election.cc:302] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:28.449743 24016 raft_consensus.cc:482] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:28.463765 23979 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:28.467347 23765 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:28.514639 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 52275.2us
I20240506 15:15:28.515601 24042 raft_consensus.cc:482] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.521521 23585 raft_consensus.cc:2781] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:28.522014 23585 raft_consensus.cc:686] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:28.523218 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.523881 21031 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:28.511319 23999 leader_election.cc:288] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:28.532858 20923 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.011s sys 0.010s
I20240506 15:15:28.516530 23770 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.536423 20996 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:28.534334 20923 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.011s sys 0.010s
W20240506 15:15:28.530511 20995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.008s sys 0.013s
W20240506 15:15:28.538168 20995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.009s sys 0.013s
I20240506 15:15:28.482256 23662 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.540095 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.509855 23834 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.562541 23585 consensus_queue.cc:237] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.580417 23834 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.580590 24049 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.581259 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.581995 21030 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:28.581468 24049 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:28.584368 24050 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:28.580451 23981 raft_consensus.cc:482] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:28.586616 23981 raft_consensus.cc:504] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:28.588421 24015 log.cc:924] Time spent T 809ee3c2d683400da1f7a753fea4a4a3 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240506 15:15:28.588670 20955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35810 (request call id 1913) took 1756 ms. Trace:
I20240506 15:15:28.589570 23857 consensus_queue.cc:1035] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240506 15:15:28.589116 20955 rpcz_store.cc:276] 0506 15:15:26.832184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:26.832403 (+ 219us) service_pool.cc:225] Handling call
0506 15:15:27.666465 (+834062us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b58000a2998. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630b20bb51 00007f630b38d3b1 00007f630b38d262 00007f631427f889 00007f63131bc3c0 00007f63132823ab 00007f6313282431 00007f6313246050 00007f6313231599 00007f6313231042 00007f630c913694
0506 15:15:27.666660 (+ 195us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:27.677985 (+ 11325us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:27.678015 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:27.728676 (+ 50661us) log.cc:842] Serialized 564 byte log entry
0506 15:15:28.409324 (+680648us) raft_consensus.cc:1631] Marking committed up to 4
0506 15:15:28.409365 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:28.409441 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:28.588311 (+178870us) raft_consensus.cc:1671] finished
0506 15:15:28.588340 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:28.588586 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12571392,"thread_start_us":496075,"threads_started":1,"wal-append.queue_time_us":712349}
I20240506 15:15:28.531900 24042 raft_consensus.cc:504] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.591152 23981 leader_election.cc:288] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:28.592788 24042 leader_election.cc:288] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:28.594370 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 58034.5us
I20240506 15:15:28.515134 24016 raft_consensus.cc:504] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.531653 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:28.583969 24001 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.606432 24047 raft_consensus.cc:2781] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:28.608232 24037 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.629182 23770 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.642258 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:28.625275 23979 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:28.643283 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.646286 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.609622 24050 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.632447 20955 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.647310 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.655303 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:28.660454 20954 raft_consensus.cc:2354] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:28.661387 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.661988 20954 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:28.663074 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:28.610028 24001 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.663784 20879 raft_consensus.cc:3037] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 8
I20240506 15:15:28.664364 20949 raft_consensus.cc:2379] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 5: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:28.672974 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.667466 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c136a9dc4474897d6b8a669145139" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.667651 20953 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:28.668144 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.668488 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:28.668582 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.669852 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.655465 23770 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.670876 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.677372 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:28.671821 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:28.674954 20882 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:28.678035 20880 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:28.690749 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.693112 20877 raft_consensus.cc:2379] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 5: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:28.675850 20952 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:28.700078 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:28.696684 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.655598 23765 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.655357 23999 raft_consensus.cc:482] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.709230 21030 raft_consensus.cc:2354] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:28.655274 24047 raft_consensus.cc:686] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:28.710831 20954 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:28.737027 20957 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240506 15:15:28.712080 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.683427 20876 raft_consensus.cc:2932] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:28.677018 24040 raft_consensus.cc:2726] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:28.743484 21023 raft_consensus.cc:2396] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20240506 15:15:28.744374 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 213587us
I20240506 15:15:28.746948 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:28.748080 21025 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 13: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:28.749169 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.649602 24037 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.700542 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0693482e35a64dfc90d3944c134718da"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:28.760565 20953 raft_consensus.cc:595] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:28.738430 20880 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
W20240506 15:15:28.757804 24057 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.136s user 0.001s sys 0.001s
I20240506 15:15:28.730093 23975 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:28.766568 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:28.782013 20877 raft_consensus.cc:3037] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Advancing to term 13
W20240506 15:15:28.760133 24023 consensus_meta.cc:287] T 6d096daea6fa4799a608705b3dc732db P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.642s user 0.007s sys 0.005s
I20240506 15:15:28.791918 24007 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:28.793023 24058 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20240506 15:15:28.793394 23984 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.793972 23984 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:28.795145 24056 log.cc:924] Time spent T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
W20240506 15:15:28.796188 24065 log.cc:924] Time spent T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240506 15:15:28.793550 24000 raft_consensus.cc:979] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:28.792600 24007 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.684207 23979 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.744587 24049 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:28.768591 24048 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:28.818517 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:28.818720 23998 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.818866 24048 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.819311 20953 raft_consensus.cc:595] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:28.819826 20953 raft_consensus.cc:603] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240506 15:15:28.819381 23998 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:28.821084 24048 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.821439 23998 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:28.671476 24050 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.741096 23765 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:28.743898 23999 raft_consensus.cc:504] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:28.827008 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 575896us
I20240506 15:15:28.801251 23662 leader_election.cc:288] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 13 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.796140 23984 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:28.842470 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.845249 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.813973 24040 raft_consensus.cc:1067] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Signalling peer 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:28.833724 20956 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 8.
I20240506 15:15:28.848951 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.857113 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:28.798575 23975 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.857826 20953 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:28.859283 23975 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.860992 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629a891c5ba546aeb0e0027a02f1b955" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.861517 23999 leader_election.cc:288] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:28.862284 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.862370 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.863659 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.863905 21023 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:28.864339 21024 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:28.864754 23984 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:28.865345 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89eb38afe3b43719809589e0aeb7eef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.865233 23984 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.866730 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.846221 24016 leader_election.cc:288] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.882966 20879 raft_consensus.cc:2454] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:28.887194 24016 raft_consensus.cc:482] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.887797 24016 raft_consensus.cc:504] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.889972 24016 leader_election.cc:288] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.867573 23984 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.892654 23984 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:28.893121 21029 raft_consensus.cc:1261] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240506 15:15:28.894809 24037 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.895162 23765 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:28.896157 24063 log.cc:924] Time spent T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20240506 15:15:28.887955 21030 raft_consensus.cc:1261] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:28.899137 20877 raft_consensus.cc:2454] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 13.
I20240506 15:15:28.900225 21031 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 5.
I20240506 15:15:28.902410 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.905611 21031 raft_consensus.cc:1261] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20240506 15:15:28.897179 24057 log.cc:924] Time spent T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240506 15:15:28.901880 23884 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:28.928844 23884 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.931707 23884 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.932431 24016 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.933209 24016 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:28.939711 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 406060us
I20240506 15:15:28.941452 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:28.942343 21031 raft_consensus.cc:3032] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:28.942728 21031 raft_consensus.cc:729] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:28.936017 24016 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.893205 23984 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:28.946346 21031 consensus_queue.cc:260] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.965112 20846 leader_election.cc:302] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:28.968617 21031 raft_consensus.cc:3037] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:28.970506 20846 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:28.970737 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:28.944005 24050 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.974423 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.870312 24047 consensus_queue.cc:237] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:28.957438 24001 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:28.961549 23770 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:28.956841 24049 raft_consensus.cc:482] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:28.996140 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea5e32ab06245dfb58c1f9b00c7fc17" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:28.992554 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.004102 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.006237 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9c7ee4280049d091aea89be873b5c8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.019095 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.021076 20953 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:29.003212 23834 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:28.972532 23837 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240506 15:15:29.029044 21031 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
I20240506 15:15:29.016868 24049 raft_consensus.cc:504] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.028424 24007 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.036032 23884 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.056726 23770 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.066792 23979 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:29.081558 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 297448us
I20240506 15:15:29.086982 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.087819 20953 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 13: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:29.090663 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.091578 20957 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:29.107726 23585 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.080722 23803 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.075305 23884 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.118028 20955 raft_consensus.cc:1226] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:29.122202 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.123250 20953 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 5.
I20240506 15:15:29.078907 23770 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.129874 20845 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:29.116235 23585 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.133756 20845 leader_election.cc:398] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
I20240506 15:15:29.136102 20845 leader_election.cc:401] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20240506 15:15:29.134244 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 348306us
I20240506 15:15:29.139305 20998 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.161223 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.161975 20955 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:29.138720 20953 raft_consensus.cc:2932] T a9f4d07ea30646d0805b63bb641358be P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:29.164005 20998 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.167960 20998 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:29.180596 23770 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:29.181957 24030 consensus_meta.cc:287] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 1.112s user 0.006s sys 0.008s
I20240506 15:15:29.131093 23984 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.188268 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.133642 23981 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:29.189033 20955 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:29.115607 23803 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:29.190551 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 197630us
I20240506 15:15:29.195552 23585 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.180712 24049 leader_election.cc:288] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.177070 23975 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:29.170987 24035 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.172829 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.197603 24049 raft_consensus.cc:3037] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 7
I20240506 15:15:29.188555 23993 raft_consensus.cc:2726] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240506 15:15:29.188910 23662 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:29.218158 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.190997 23981 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 15:15:29.197876 24071 log.cc:924] Time spent T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20240506 15:15:29.219134 20955 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:29.200627 23975 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
W20240506 15:15:29.256831 24075 log.cc:924] Time spent T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240506 15:15:29.215214 24035 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.201321 20998 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.221637 23662 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:29.229348 23981 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
W20240506 15:15:29.269323 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 68118.3us
W20240506 15:15:29.270268 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 139778us
I20240506 15:15:29.246323 23884 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:29.266166 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 96202.8us
I20240506 15:15:29.248833 23837 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.260650 23975 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:29.275146 20998 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:29.269013 23662 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:29.269364 24042 raft_consensus.cc:2726] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:29.308642 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.304548 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.305768 24079 raft_consensus.cc:2781] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 13 FOLLOWER]: Leader election won for term 13
I20240506 15:15:29.313387 23837 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.272861 24077 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:29.309373 20953 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
W20240506 15:15:29.324935 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 120344us
I20240506 15:15:29.307775 24086 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:29.346746 23837 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:29.351477 24072 log.cc:924] Time spent T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20240506 15:15:29.353288 23803 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.324398 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84d513d60b64cba9c6bc3f8077e6b73" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:15:29.356829 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 51827.6us
I20240506 15:15:29.363791 20996 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.335896 24079 raft_consensus.cc:686] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [term 13 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:29.343622 24035 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:29.383634 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 92226.2us
W20240506 15:15:29.384594 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 79826.2us
I20240506 15:15:29.391372 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.390161 20845 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:29.382511 24001 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:29.404150 21025 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:29.412595 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:15:29.415107 24088 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240506 15:15:29.400596 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:29.419914 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 90389.6us
I20240506 15:15:29.420482 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.421945 21027 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:29.422224 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.424365 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.426899 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.427187 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.428042 21023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0693482e35a64dfc90d3944c134718da"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:47952
I20240506 15:15:29.428658 21023 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:29.429004 21023 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:29.429811 21031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "208b7cbf36514c419696de9c14606c16"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:37012
I20240506 15:15:29.434413 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.402170 24001 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.395959 24093 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.402246 23998 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:29.430392 21031 raft_consensus.cc:595] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:29.437230 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.440495 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.443315 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.442637 20846 leader_election.cc:398] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:29.444058 20846 leader_election.cc:401] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240506 15:15:29.433256 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:29.355123 24086 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.445899 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.447355 21029 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:29.435603 24048 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:29.441347 24001 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:29.452013 21030 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:29.441632 23998 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.456419 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.430706 24096 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20240506 15:15:29.450052 24031 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:29.469660 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.453121 24093 raft_consensus.cc:2726] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:29.451237 24048 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:29.465600 21023 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.448447 24100 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:29.467880 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.451614 24098 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:29.454044 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.456301 23998 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:29.471541 24030 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:29.468662 24031 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.479146 24048 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:29.493511 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.494612 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.482307 24098 raft_consensus.cc:482] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.496878 21023 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.480422 24100 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:29.499394 24100 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:29.485258 24093 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240506 15:15:29.454603 24079 consensus_queue.cc:237] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.485232 24049 raft_consensus.cc:2726] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913) with higher term. Message: Invalid argument: T 2719acf5d3614c8bb6f5d84b1bc791de P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7.
W20240506 15:15:29.505108 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 51563.8us
I20240506 15:15:29.493887 24031 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:29.482532 24087 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20240506 15:15:29.509924 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.509218 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.507797 20846 leader_election.cc:302] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.487550 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9178d1bf59624e7f8948c33552e74a00" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:29.491219 24030 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:29.515064 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.518553 20957 raft_consensus.cc:3032] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:29.516278 20845 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.518996 20957 raft_consensus.cc:729] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:29.516837 20846 leader_election.cc:302] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.525104 20846 leader_election.cc:302] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.537516 24096 raft_consensus.cc:2726] T 629a891c5ba546aeb0e0027a02f1b955 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:29.582974 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.504818 24098 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:29.554316 24051 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:29.556092 24086 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.558408 23975 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.559537 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.520196 24030 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:29.550298 24102 consensus_queue.cc:1035] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240506 15:15:29.608425 20958 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:29.608083 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:29.582360 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 57363.1us
I20240506 15:15:29.518810 24049 raft_consensus.cc:2726] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.541370 24106 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:29.591091 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.591658 20845 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:29.592701 21027 raft_consensus.cc:1226] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:29.597304 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.542573 20957 consensus_queue.cc:260] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.619727 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "286345f896bc43ec975183896aee147c"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:29.523910 24093 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:29.610383 20846 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.602630 24095 raft_consensus.cc:2726] T c12c136a9dc4474897d6b8a669145139 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:29.624209 20846 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.611555 21654 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.613979 21024 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:29.620334 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:29.626832 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.627069 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.627422 20921 leader_election.cc:302] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:29.607393 24003 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.625808 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.629379 20957 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:29.632973 24016 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:29.633100 24080 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.633281 23984 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:29.633287 24092 raft_consensus.cc:2726] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:29.635411 21029 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:29.640431 24049 consensus_queue.cc:1046] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 15:15:29.633685 23770 raft_consensus.cc:2726] T 9178d1bf59624e7f8948c33552e74a00 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:29.660701 20953 raft_consensus.cc:595] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Received request to transfer leadership to TS 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:29.628425 24046 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:29.607439 24051 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.697166 20921 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:29.661835 24042 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:29.649806 23662 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.633636 23937 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.048s
I20240506 15:15:29.652693 23834 raft_consensus.cc:482] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.671347 24046 raft_consensus.cc:482] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.668465 23975 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.652915 24007 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240506 15:15:29.656311 24095 consensus_queue.cc:1035] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20240506 15:15:29.657706 23984 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20240506 15:15:29.676503 24102 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.719012 21030 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:29.719231 23662 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.653376 24003 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.706938 24049 consensus_queue.cc:1225] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer 7610624a16b44896b6aacea1928ab20e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240506 15:15:29.715155 23765 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:29.716081 24016 raft_consensus.cc:686] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:29.721493 24046 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:29.720523 24094 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20240506 15:15:29.753103 24093 raft_consensus.cc:686] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:29.794401 24080 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.777323 20957 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
W20240506 15:15:29.782382 23981 consensus_meta.cc:287] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.501s user 0.011s sys 0.000s
I20240506 15:15:29.808517 23981 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.810488 23981 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.744982 23834 raft_consensus.cc:504] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.812781 24087 raft_consensus.cc:3032] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:29.813198 24087 raft_consensus.cc:729] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:29.813445 23834 leader_election.cc:288] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.806371 24048 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.781055 24003 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.813997 24087 consensus_queue.cc:260] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.815240 24087 raft_consensus.cc:3037] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:29.815527 24048 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.776136 23857 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.810513 23998 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.817502 24080 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.817399 23857 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.819140 23998 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.819612 21023 raft_consensus.cc:1226] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:29.776639 23979 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.821697 24100 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.822016 23979 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.822886 24036 consensus_queue.cc:1046] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:29.823906 24100 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.824290 24036 raft_consensus.cc:3032] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:29.824800 24036 raft_consensus.cc:729] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:29.825007 23979 leader_election.cc:288] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.771834 24098 raft_consensus.cc:504] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.826033 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.825877 24036 consensus_queue.cc:260] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.827355 24098 leader_election.cc:288] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.757059 24102 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.788002 24031 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.829838 24102 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.829977 21031 raft_consensus.cc:1226] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 10. Current term is 11. Ops: []
I20240506 15:15:29.772933 24051 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.772105 24016 consensus_queue.cc:237] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.833074 24031 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.759246 24114 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.834667 24114 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.826735 21023 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 5.
I20240506 15:15:29.837332 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:29.838786 24086 consensus_queue.cc:1046] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240506 15:15:29.765866 24001 raft_consensus.cc:504] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.840729 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:29.841697 21026 raft_consensus.cc:2379] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:29.844419 21024 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:29.844794 24086 raft_consensus.cc:3032] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Stepping down as leader of term 10
I20240506 15:15:29.845185 24086 raft_consensus.cc:729] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:29.845992 24086 consensus_queue.cc:260] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.847445 24086 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Advancing to term 11
W20240506 15:15:29.848820 21654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 44340.9us
I20240506 15:15:29.849045 24001 leader_election.cc:288] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:29.819670 24080 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.843465 24114 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:29.864068 24111 log.cc:924] Time spent T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20240506 15:15:29.869851 24093 consensus_queue.cc:237] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:29.872503 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 247138us
I20240506 15:15:29.873222 21653 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:29.873694 20846 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.884498 20846 leader_election.cc:302] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.887817 20846 leader_election.cc:302] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.889391 20846 leader_election.cc:302] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.892340 24049 consensus_queue.cc:1035] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:29.888990 23834 raft_consensus.cc:2726] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:29.893613 23857 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.894060 23981 raft_consensus.cc:2726] T 3ea5e32ab06245dfb58c1f9b00c7fc17 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.894522 24049 raft_consensus.cc:2726] T 4c9c7ee4280049d091aea89be873b5c8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.889142 24102 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.895592 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.897866 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.899158 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.900156 21653 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:29.886299 24001 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:29.901685 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.900959 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.903911 23857 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:29.904489 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2719acf5d3614c8bb6f5d84b1bc791de" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:29.909426 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.889958 24086 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:29.911383 20880 raft_consensus.cc:1261] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:29.915227 20879 raft_consensus.cc:2454] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
W20240506 15:15:29.918581 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 213403us
I20240506 15:15:29.919215 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.920341 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.888465 24030 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:29.912647 24086 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.924549 20921 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.929458 21030 raft_consensus.cc:1226] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 10. Current term is 11. Ops: []
I20240506 15:15:29.927784 20921 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.904907 24001 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.935427 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.934000 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.934892 20996 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:29.937043 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.943236 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.927393 24086 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:29.944617 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.946341 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:29.932621 24114 raft_consensus.cc:2781] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:29.946300 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:29.947186 20879 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:29.948558 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.950985 20953 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 10: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:29.923996 24046 raft_consensus.cc:504] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.954893 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.961848 20879 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:29.958312 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:29.949494 24122 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:29.963766 24122 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:29.965955 24122 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.960942 20950 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:29.936641 24121 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:29.957947 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:29.967407 24121 raft_consensus.cc:482] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.967823 24121 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:29.967938 20875 raft_consensus.cc:2354] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 10. Current term is 11.
I20240506 15:15:29.962669 20882 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 13: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:29.962739 20874 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:29.962939 20880 raft_consensus.cc:2354] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:29.954339 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:29.969798 20877 raft_consensus.cc:3037] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:29.947614 24100 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election won for term 10
I20240506 15:15:29.970623 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:29.973416 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.947952 24114 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:29.987534 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:29.975039 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.975350 24030 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:29.977469 24036 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:29.978420 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.979120 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:29.979164 24046 leader_election.cc:288] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:29.977499 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.013080 20951 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:29.980388 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.980123 23857 consensus_queue.cc:1046] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240506 15:15:29.981174 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.013834 20882 raft_consensus.cc:2379] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 11: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:29.985216 20954 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:29.985247 20952 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:29.988608 20883 raft_consensus.cc:1261] T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:29.993167 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.015751 20881 raft_consensus.cc:2379] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 13: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:30.016088 20950 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:30.008529 20779 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39518 (request call id 87) took 2568 ms. Trace:
I20240506 15:15:30.017647 20779 rpcz_store.cc:276] 0506 15:15:27.440383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:27.440568 (+ 185us) service_pool.cc:225] Handling call
0506 15:15:30.008468 (+2567900us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 15:15:28.691994 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 15:15:28.828139 (+136145us) write_op.cc:432] Acquiring schema lock in shared mode
0506 15:15:28.835371 (+ 7232us) write_op.cc:435] Acquired schema lock
0506 15:15:28.845927 (+ 10556us) tablet.cc:660] Decoding operations
0506 15:15:28.914665 (+ 68738us) write_op.cc:620] Acquiring the partition lock for write op
0506 15:15:28.938303 (+ 23638us) write_op.cc:641] Partition lock acquired for write op
0506 15:15:28.940690 (+ 2387us) tablet.cc:683] Acquiring locks for 4 operations
0506 15:15:29.009545 (+ 68855us) tablet.cc:699] Row locks acquired
0506 15:15:29.009584 (+ 39us) write_op.cc:260] PREPARE: finished
0506 15:15:29.027756 (+ 18172us) write_op.cc:270] Start()
0506 15:15:29.030860 (+ 3104us) write_op.cc:276] Timestamp: P: 1715008529026235 usec, L: 0
0506 15:15:29.031975 (+ 1115us) op_driver.cc:331] REPLICATION: starting
0506 15:15:29.096395 (+ 64420us) log.cc:842] Serialized 2063 byte log entry
0506 15:15:29.721029 (+624634us) op_driver.cc:444] REPLICATION: finished
0506 15:15:29.933621 (+212592us) write_op.cc:301] APPLY: starting
0506 15:15:29.940795 (+ 7174us) tablet.cc:1365] starting BulkCheckPresence
0506 15:15:29.948372 (+ 7577us) tablet.cc:1368] finished BulkCheckPresence
0506 15:15:29.948501 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 15:15:29.949110 (+ 609us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 15:15:29.949318 (+ 208us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0506 15:15:29.949370 (+ 52us) write_op.cc:312] APPLY: finished
0506 15:15:29.974633 (+ 25263us) log.cc:842] Serialized 41 byte log entry
0506 15:15:29.974804 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:29.974920 (+ 116us) write_op.cc:454] Released schema lock
0506 15:15:29.974961 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0506 15:15:29.975229 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0506 15:15:29.975258 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":170446,"apply.run_cpu_time_us":2857,"apply.run_wall_time_us":46325,"num_ops":4,"prepare.queue_time_us":287634,"prepare.run_cpu_time_us":19907,"prepare.run_wall_time_us":640949,"raft.queue_time_us":240520,"raft.run_cpu_time_us":58286,"raft.run_wall_time_us":211606,"replication_time_us":684287,"thread_start_us":658302,"threads_started":4,"wal-append.queue_time_us":252200}]]}
I20240506 15:15:30.020897 20955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
mode: GRACEFUL
new_leader_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:34038
I20240506 15:15:30.021484 20955 raft_consensus.cc:595] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Received request to transfer leadership to TS 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:30.012032 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:29.994694 24114 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:30.009011 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.024276 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.025287 20877 raft_consensus.cc:2454] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:30.025182 20881 raft_consensus.cc:2454] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:30.032676 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:30.036590 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:30.037333 21023 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:30.038190 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.040004 20877 raft_consensus.cc:1261] T 2719acf5d3614c8bb6f5d84b1bc791de P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240506 15:15:30.045011 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.052770 23765 raft_consensus.cc:979] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:30.053390 23765 raft_consensus.cc:1067] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Signalling peer 7610624a16b44896b6aacea1928ab20e to start an election
I20240506 15:15:30.056577 24087 raft_consensus.cc:482] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:30.057200 24087 raft_consensus.cc:504] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:30.058707 24001 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.020823 24100 raft_consensus.cc:686] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [term 10 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:30.061089 24100 consensus_queue.cc:237] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.012712 24031 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.050s
I20240506 15:15:30.064635 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.065214 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.065850 19726 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240506 15:15:30.065969 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.067703 19726 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240506 15:15:30.068801 24099 raft_consensus.cc:3032] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Stepping down as leader of term 10
I20240506 15:15:30.069259 24099 raft_consensus.cc:729] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:30.070191 24099 consensus_queue.cc:260] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:30.070859 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.064764 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.075858 23803 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:30.076754 21030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "286345f896bc43ec975183896aee147c"
dest_uuid: "7610624a16b44896b6aacea1928ab20e"
from {username='slave'} at 127.0.0.1:47952
I20240506 15:15:30.076589 23803 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.077358 21030 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:30.077754 21030 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:30.078763 23803 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.079592 24087 leader_election.cc:288] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:30.069136 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.071547 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.080327 20881 raft_consensus.cc:2454] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 5.
I20240506 15:15:30.081372 20996 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.081970 24001 raft_consensus.cc:2726] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:30.082842 20996 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.083215 24001 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:30.083654 24001 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:30.083922 24035 raft_consensus.cc:2726] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:30.068847 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.086763 20952 raft_consensus.cc:2454] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:30.090361 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.091120 20956 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:30.043980 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.091991 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.071640 20876 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:30.092888 19726 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240506 15:15:30.109587 21023 raft_consensus.cc:2454] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 6.
I20240506 15:15:30.113653 20950 raft_consensus.cc:2454] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:30.084007 24001 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:30.158761 20951 raft_consensus.cc:2454] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:30.120522 20955 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 13 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240506 15:15:30.165402 20996 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.170564 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.171819 20921 leader_election.cc:302] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.171237 20951 raft_consensus.cc:2454] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:30.179934 20996 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.108836 21030 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.181217 20882 raft_consensus.cc:1261] T 805cea534ce0498f89b8551ba0b7e5d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240506 15:15:30.181334 20996 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:30.139506 24125 log.cc:924] Time spent T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.062s user 0.002s sys 0.000s
I20240506 15:15:30.180553 24122 raft_consensus.cc:2781] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:30.183303 21030 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.183598 24122 raft_consensus.cc:686] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:30.184707 24122 consensus_queue.cc:237] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.188982 23993 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:30.189477 23993 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:30.157487 20780 catalog_manager.cc:5484] T 8884787c4c4b474685c7e903e8789e90 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 6 to 8, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.191046 23662 raft_consensus.cc:2781] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader election won for term 7
I20240506 15:15:30.191648 23662 raft_consensus.cc:686] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:30.193436 20780 catalog_manager.cc:5484] T 1a88ec9e65b24b64be49384639f4f9f9 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 8 to 9, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 9 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.208880 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.210243 20846 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.216943 21030 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:30.217489 24087 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Leader election won for term 5
I20240506 15:15:30.217981 24087 raft_consensus.cc:686] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:30.218369 20846 leader_election.cc:302] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.218820 24087 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:30.209622 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.219468 24093 raft_consensus.cc:2726] T d84d513d60b64cba9c6bc3f8077e6b73 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:30.219514 24051 consensus_queue.cc:1046] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240506 15:15:30.204680 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.217078 23662 consensus_queue.cc:237] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.220062 20879 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:30.210055 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.202889 24121 raft_consensus.cc:504] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.221029 24046 raft_consensus.cc:3032] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:30.210327 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.189891 23993 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:30.071607 24099 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:30.224848 20780 catalog_manager.cc:5484] T 9b0fa8a9db084a51a74caa1382efc1c5 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 5 to 6, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 6 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
W20240506 15:15:30.233367 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 168251us
I20240506 15:15:30.234055 20780 catalog_manager.cc:5484] T c12c136a9dc4474897d6b8a669145139 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ecededcd854842d4867fda9fa63a4b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } }
I20240506 15:15:30.206446 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.236009 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a4c20ff49de9f94f4e07d3269ef" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.212613 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.238473 20996 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.238602 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.240998 20955 raft_consensus.cc:1203] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
I20240506 15:15:30.241665 20957 raft_consensus.cc:2354] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:30.241842 20996 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.242791 24035 raft_consensus.cc:2781] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:30.243166 24035 raft_consensus.cc:482] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:30.243896 24035 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:30.235956 20780 catalog_manager.cc:5484] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.243162 24003 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:30.245167 20780 catalog_manager.cc:5484] T 815d66b0c2c24db2aa336b62f17837a1 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 5 to 8, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 8 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.230381 24046 raft_consensus.cc:729] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:30.249769 24046 consensus_queue.cc:260] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.251047 24046 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:30.253182 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.254830 20879 raft_consensus.cc:1261] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240506 15:15:30.255213 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.255604 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.256760 20955 raft_consensus.cc:1226] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240506 15:15:30.258556 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.260375 20883 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:30.263983 24031 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:30.264650 24031 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.266701 24031 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.269623 23766 raft_consensus.cc:482] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:30.270185 23766 raft_consensus.cc:504] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.230471 24079 raft_consensus.cc:2781] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election won for term 8
I20240506 15:15:30.272631 24079 raft_consensus.cc:686] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:30.124454 24086 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:30.273723 23766 leader_election.cc:288] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.278268 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.285738 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.246760 20780 catalog_manager.cc:5484] T 809ee3c2d683400da1f7a753fea4a4a3 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 6 to 7, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.293298 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:30.294184 21026 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:30.299851 20780 catalog_manager.cc:5484] T dd8ce6e0da3e465f875171ac339f3b50 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 6 to 7, leader changed from 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 7 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.303023 20780 catalog_manager.cc:5484] T 35b78f712a194a51abc96c8500a9a611 P 7610624a16b44896b6aacea1928ab20e reported cstate change: term changed from 9 to 12, leader changed from ecededcd854842d4867fda9fa63a4b66 (127.19.67.130) to 7610624a16b44896b6aacea1928ab20e (127.19.67.131). New cstate: current_term: 12 leader_uuid: "7610624a16b44896b6aacea1928ab20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240506 15:15:30.304440 20996 leader_election.cc:398] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:30.298162 20954 raft_consensus.cc:2932] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } } }
I20240506 15:15:30.230919 24114 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.304790 20996 leader_election.cc:401] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240506 15:15:30.309933 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7268b79c9094189823e8d245b8c579a" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.312332 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.330521 20956 raft_consensus.cc:2354] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:30.312750 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.313098 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:30.314209 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca08769950c4ef2bae0df854c025154" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.274286 24030 raft_consensus.cc:2781] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:30.315382 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.336653 20950 raft_consensus.cc:2362] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9. Re-sending same reply.
I20240506 15:15:30.316165 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81525cec16e140db99cd0ef9f2717778" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.331830 20952 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9. Ops: []
I20240506 15:15:30.288195 24086 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:30.352459 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 48130us
I20240506 15:15:30.359788 23937 raft_consensus.cc:3037] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:30.273660 24079 consensus_queue.cc:237] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.355118 24030 raft_consensus.cc:686] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 9 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:30.391929 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:30.393456 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:30.394063 21027 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
W20240506 15:15:30.388144 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 70930.3us
I20240506 15:15:30.408623 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:30.415733 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:30.417732 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:30.419004 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.419271 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.419648 20950 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:30.421963 20845 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.422791 21026 raft_consensus.cc:2454] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:15:30.424217 21653 leader_election.cc:302] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:30.425171 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f44ddbf0044d1c9c94508ba5aa2406" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:30.425526 24114 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:30.428098 23993 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.428540 24001 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.429812 21031 raft_consensus.cc:1226] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240506 15:15:30.430824 24001 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.431864 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.433159 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.434331 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801b36bb14184241921c1b97377314fb" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.434933 20958 raft_consensus.cc:3037] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:30.436751 20996 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.438500 24003 consensus_queue.cc:1035] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240506 15:15:30.446017 24003 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:30.446384 24003 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting leader election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:30.446756 24003 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:30.447105 20949 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:30.440413 23998 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:30.441030 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.439335 20998 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:30.442517 24036 raft_consensus.cc:2781] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:30.450402 21027 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 11.
I20240506 15:15:30.450927 20949 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:30.450845 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.452169 20953 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 9: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:30.443586 24121 leader_election.cc:288] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:30.444618 21685 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 6.
I20240506 15:15:30.447520 21023 raft_consensus.cc:1226] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:30.370169 24030 consensus_queue.cc:237] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.384591 24100 raft_consensus.cc:482] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:30.448124 20951 raft_consensus.cc:2354] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10.
I20240506 15:15:30.457347 20846 leader_election.cc:302] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.464586 20958 raft_consensus.cc:2454] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 11.
I20240506 15:15:30.479257 20950 raft_consensus.cc:1226] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20240506 15:15:30.465865 20924 leader_election.cc:334] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145): Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.889s (SENT)
I20240506 15:15:30.472641 20845 leader_election.cc:302] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:30.471608 20998 leader_election.cc:398] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 10. Current term is 11.
I20240506 15:15:30.480648 20998 leader_election.cc:401] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240506 15:15:30.480481 20924 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.492327 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.493486 20846 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.491475 20957 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:30.504283 20846 leader_election.cc:302] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.527812 21032 raft_consensus.cc:1226] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:30.503387 20998 leader_election.cc:398] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:30.526875 24093 consensus_queue.cc:1046] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20240506 15:15:30.470903 24120 raft_consensus.cc:2726] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 15:15:30.533612 24031 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145) with higher term. Message: Invalid argument: T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 10. Current term is 11.
I20240506 15:15:30.526562 24118 raft_consensus.cc:2726] T e4ed6a4c20ff49de9f94f4e07d3269ef P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:30.533197 24087 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240506 15:15:30.470352 24036 raft_consensus.cc:686] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Leader. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
I20240506 15:15:30.527297 24046 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:30.466387 23998 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:30.546454 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 80612.9us
I20240506 15:15:30.479635 23766 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:30.570876 24036 consensus_queue.cc:237] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:30.577754 24128 log.cc:924] Time spent T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20240506 15:15:30.540031 23993 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.584270 23998 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:30.585225 24118 raft_consensus.cc:2726] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:30.585498 24120 raft_consensus.cc:2726] T 48f44ddbf0044d1c9c94508ba5aa2406 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:30.591029 20924 leader_election.cc:302] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.594024 20924 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.597867 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:30.606222 20924 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.494056 24100 raft_consensus.cc:504] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.596472 24121 raft_consensus.cc:2726] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:30.596711 24046 raft_consensus.cc:2726] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:30.608943 23884 raft_consensus.cc:2781] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 15:15:30.609218 23770 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:30.647814 21653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 223712us
I20240506 15:15:30.642710 23884 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
W20240506 15:15:30.695034 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 201358us
W20240506 15:15:30.696326 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 192151us
I20240506 15:15:30.697149 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:30.697628 20846 leader_election.cc:398] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8.
I20240506 15:15:30.698398 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:30.699040 20953 raft_consensus.cc:2454] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
W20240506 15:15:30.699676 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 227112us
W20240506 15:15:30.700839 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 92570.2us
W20240506 15:15:30.702416 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 283193us
W20240506 15:15:30.702822 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 173722us
I20240506 15:15:30.706982 20924 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.710935 23937 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:30.691591 23884 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:30.729378 21029 raft_consensus.cc:1226] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [term 9 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 8. Current term is 9. Ops: []
W20240506 15:15:30.746512 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 47334.3us
W20240506 15:15:30.747556 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 44097.9us
I20240506 15:15:30.748003 20846 leader_election.cc:302] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:30.760128 20883 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:30.760272 21030 raft_consensus.cc:2454] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
W20240506 15:15:30.764405 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 61325.6us
W20240506 15:15:30.764935 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 64369.8us
I20240506 15:15:30.765601 20845 leader_election.cc:302] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:30.713872 24051 consensus_queue.cc:1035] T 2719acf5d3614c8bb6f5d84b1bc791de P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240506 15:15:30.794421 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.794632 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b7301301694836b2d08bf22e3f2e38" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:30.795262 20883 raft_consensus.cc:2379] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 8: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
W20240506 15:15:30.796275 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45950.2us
I20240506 15:15:30.798408 21026 raft_consensus.cc:1226] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:30.804332 21032 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:30.752489 24093 consensus_queue.cc:1046] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20240506 15:15:30.808141 21031 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20240506 15:15:30.826453 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 78554.7us
I20240506 15:15:30.803339 24094 consensus_queue.cc:1046] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20240506 15:15:30.816013 24100 leader_election.cc:288] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
W20240506 15:15:30.826953 24140 log.cc:924] Time spent T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20240506 15:15:30.827147 24159 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:30.808531 24087 consensus_queue.cc:1046] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20240506 15:15:30.878914 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 108826us
W20240506 15:15:30.885715 24142 log.cc:924] Time spent T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
W20240506 15:15:30.886672 24139 log.cc:924] Time spent T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20240506 15:15:30.827265 24151 raft_consensus.cc:2781] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader election won for term 11
I20240506 15:15:30.893014 24016 consensus_queue.cc:1046] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:30.862370 24093 raft_consensus.cc:3032] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Stepping down as leader of term 8
I20240506 15:15:30.852891 24152 raft_consensus.cc:2726] T 3ca08769950c4ef2bae0df854c025154 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:30.899464 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 77660.4us
I20240506 15:15:30.900846 20998 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:30.902230 20998 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:30.907559 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 164343us
W20240506 15:15:30.894286 24135 log.cc:924] Time spent T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
I20240506 15:15:30.818673 24143 consensus_queue.cc:1035] T f5a9099e10624a769e45882e9b864930 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.142s
W20240506 15:15:30.909312 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 49846.6us
W20240506 15:15:30.907590 24088 log.cc:924] Time spent T 00000000000000000000000000000000 P 785bf14009d340bcbb6be5e678b5f9cf: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20240506 15:15:30.917469 24003 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.908835 20924 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.913086 24048 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240506 15:15:30.920603 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 103558us
I20240506 15:15:30.826773 24146 consensus_queue.cc:1046] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20240506 15:15:30.898969 24093 raft_consensus.cc:729] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:30.930181 20924 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.935322 20924 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:30.907964 23993 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:30.877628 24147 consensus_queue.cc:1035] T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20240506 15:15:30.943261 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 128612us
I20240506 15:15:30.944360 20924 leader_election.cc:302] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.949528 20996 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:30.921834 24159 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:30.942004 24035 consensus_meta.cc:287] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.698s user 0.013s sys 0.008s
I20240506 15:15:30.949939 24007 raft_consensus.cc:2781] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 15:15:30.948546 24154 raft_consensus.cc:2726] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:30.969231 20924 leader_election.cc:302] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.952940 23993 raft_consensus.cc:686] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [term 9 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
W20240506 15:15:30.978267 24150 log.cc:924] Time spent T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20240506 15:15:30.978166 21685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59110 (request call id 325) took 1140 ms. Trace:
I20240506 15:15:30.979820 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.956813 24051 raft_consensus.cc:2726] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240506 15:15:30.983073 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 61137.1us
I20240506 15:15:30.994486 24087 consensus_queue.cc:1225] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer 7610624a16b44896b6aacea1928ab20e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240506 15:15:30.968713 24007 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.001536 20924 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:30.957125 24035 raft_consensus.cc:504] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:30.983489 24114 raft_consensus.cc:2726] T 23b7301301694836b2d08bf22e3f2e38 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:31.004122 24167 consensus_queue.cc:1035] T 805cea534ce0498f89b8551ba0b7e5d8 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240506 15:15:31.000463 24166 raft_consensus.cc:3032] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:30.982295 24152 raft_consensus.cc:3032] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:31.005472 23765 raft_consensus.cc:979] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:31.005321 24170 consensus_queue.cc:1046] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:31.006433 23884 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:30.981599 24029 raft_consensus.cc:2781] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:31.012636 20924 leader_election.cc:302] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.003652 24007 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:31.025507 21025 raft_consensus.cc:1226] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: []
I20240506 15:15:31.028894 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.029596 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.029817 20879 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240506 15:15:31.010756 24154 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:31.034004 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.036743 23765 raft_consensus.cc:1067] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:30.971120 24093 consensus_queue.cc:260] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.045028 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:31.048439 23993 consensus_queue.cc:237] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.017959 24051 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:31.050402 24100 raft_consensus.cc:979] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e: : Instructing follower 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:31.018297 24152 raft_consensus.cc:729] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.051000 24100 raft_consensus.cc:1067] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Signalling peer 00fe71c1675441379f5f2c7490f9e63a to start an election
I20240506 15:15:31.051858 24152 consensus_queue.cc:260] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.023229 24003 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:31.023667 24029 raft_consensus.cc:482] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:31.053670 24029 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:30.981926 21685 rpcz_store.cc:276] 0506 15:15:29.835917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:29.836963 (+ 1046us) service_pool.cc:225] Handling call
0506 15:15:30.976152 (+1139189us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54006dea54. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:30.976306 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":843960064}
I20240506 15:15:31.023056 24168 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240506 15:15:31.058900 24035 leader_election.cc:288] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:31.063701 23884 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.033757 24154 raft_consensus.cc:482] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:31.065455 24154 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:31.012416 24166 raft_consensus.cc:729] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.069258 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:31.070050 21031 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:31.071384 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:31.024761 24122 raft_consensus.cc:2726] T 35b78f712a194a51abc96c8500a9a611 P ecededcd854842d4867fda9fa63a4b66 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240506 15:15:31.068874 24166 consensus_queue.cc:260] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.078529 21031 raft_consensus.cc:1261] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240506 15:15:31.053622 24152 raft_consensus.cc:3037] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:31.019997 24151 raft_consensus.cc:686] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [term 11 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
W20240506 15:15:31.095185 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 176323us
I20240506 15:15:31.098332 20846 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:31.099342 20846 leader_election.cc:302] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:31.096791 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 143187us
I20240506 15:15:31.103870 20878 raft_consensus.cc:3032] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:31.104686 20878 raft_consensus.cc:729] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.105764 20878 consensus_queue.cc:260] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.108536 20846 leader_election.cc:398] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a for earlier term 9. Current term is 10.
W20240506 15:15:31.094298 24036 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.003s sys 0.006s
I20240506 15:15:31.115003 20954 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:31.116923 20950 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240506 15:15:31.118409 20878 raft_consensus.cc:3037] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:31.113070 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:31.123245 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.125358 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:31.125569 20996 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:31.110877 24036 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s user 0.004s sys 0.006s
I20240506 15:15:31.130930 20956 raft_consensus.cc:1261] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240506 15:15:31.109283 24146 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:31.130121 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:31.109969 24168 raft_consensus.cc:2726] T 9b0fa8a9db084a51a74caa1382efc1c5 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:31.094583 23884 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20240506 15:15:31.141515 24151 consensus_queue.cc:237] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.132691 24172 consensus_queue.cc:1046] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20240506 15:15:31.124181 24175 consensus_queue.cc:1159] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ecededcd854842d4867fda9fa63a4b66" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240506 15:15:31.127979 24173 consensus_queue.cc:1046] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240506 15:15:31.148391 23803 raft_consensus.cc:2781] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:31.143958 24007 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.124809 24143 raft_consensus.cc:482] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:31.155028 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.165092 20875 raft_consensus.cc:2379] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e in current term 8: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:31.300124 20878 raft_consensus.cc:1261] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:31.156160 24172 raft_consensus.cc:3032] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Stepping down as leader of term 9
I20240506 15:15:31.345300 24172 raft_consensus.cc:729] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
W20240506 15:15:31.345239 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 198659us
I20240506 15:15:31.152585 24036 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:15:31.359025 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 228946us
I20240506 15:15:31.377382 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.377481 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.378026 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481a6073b18543d58315ee2fe4522232" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.377697 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.378763 20879 raft_consensus.cc:2379] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 9: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:31.380205 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d66b0c2c24db2aa336b62f17837a1" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.380532 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.383124 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85358bc4d3f42d897b4c2868c83c8ec" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.383920 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81c09e0400045b180379648eed9b219" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.348160 24051 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.424819 20877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edfbe39391094dce8178f175dfb05165"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50474
I20240506 15:15:31.425802 20877 raft_consensus.cc:482] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:31.426299 20877 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:31.261392 24143 raft_consensus.cc:504] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.371325 23803 raft_consensus.cc:686] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Leader. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:31.369164 24036 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.410691 23884 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.470736 20875 raft_consensus.cc:3032] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:31.471354 20875 raft_consensus.cc:729] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.480505 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.412544 24051 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.492486 20924 leader_election.cc:302] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:31.494328 20924 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.436219 24147 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20240506 15:15:31.397271 24178 consensus_queue.cc:1035] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20240506 15:15:31.503131 21029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47948 (request call id 1719) took 1177 ms. Trace:
I20240506 15:15:31.503604 21029 rpcz_store.cc:276] 0506 15:15:30.389196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:30.389556 (+ 360us) service_pool.cc:225] Handling call
0506 15:15:30.425077 (+ 35521us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:30.729971 (+304894us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:31.496878 (+766907us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54001c20d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:31.497046 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":699374336}
I20240506 15:15:31.499236 20877 raft_consensus.cc:504] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.472419 20875 consensus_queue.cc:260] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.508374 20875 raft_consensus.cc:3037] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:31.510798 20876 raft_consensus.cc:1226] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:31.472735 23765 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.498416 23770 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 15:15:31.506705 24177 raft_consensus.cc:482] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:31.505795 24151 raft_consensus.cc:482] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:31.518129 24007 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.435719 24172 consensus_queue.cc:260] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.463565 23884 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.496287 20924 leader_election.cc:302] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.519349 20877 leader_election.cc:288] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.517302 24122 raft_consensus.cc:2726] T 481a6073b18543d58315ee2fe4522232 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240506 15:15:31.541216 20875 raft_consensus.cc:1261] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240506 15:15:31.541842 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f207fe580fd74c83b4540a552300d5b8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.556320 20875 pending_rounds.cc:76] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 3
W20240506 15:15:31.542734 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 144541us
I20240506 15:15:31.556665 20875 pending_rounds.cc:97] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240506 15:15:31.556973 20875 raft_consensus.cc:2866] T 81525cec16e140db99cd0ef9f2717778 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:31.556809 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.526396 23803 consensus_queue.cc:237] T b96eaaaebde34390bee0dafdfb156080 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.562750 20875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "208b7cbf36514c419696de9c14606c16"
dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
from {username='slave'} at 127.0.0.1:50468
I20240506 15:15:31.538758 24118 raft_consensus.cc:3032] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:31.574249 20875 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 15:15:31.579969 20875 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
W20240506 15:15:31.590194 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 94035.8us
I20240506 15:15:31.591274 20924 leader_election.cc:302] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.592756 20924 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.518635 23765 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:31.619117 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 47685us
I20240506 15:15:31.620481 20924 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:31.624200 24122 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:31.632869 24190 raft_consensus.cc:2726] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240506 15:15:31.634471 24154 consensus_meta.cc:287] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66: Time spent flushing consensus metadata: real 0.568s user 0.009s sys 0.000s
I20240506 15:15:31.634925 24154 raft_consensus.cc:504] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.579555 24118 raft_consensus.cc:729] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.636610 20955 raft_consensus.cc:2354] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
W20240506 15:15:31.636853 24188 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20240506 15:15:31.636741 24121 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P ecededcd854842d4867fda9fa63a4b66 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:31.637385 24118 consensus_queue.cc:260] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.638362 24087 raft_consensus.cc:3032] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:31.638499 24118 raft_consensus.cc:3037] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
W20240506 15:15:31.638636 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 59110.2us
I20240506 15:15:31.632635 24122 raft_consensus.cc:504] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.577978 23857 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:31.596210 24007 raft_consensus.cc:2726] T e81c09e0400045b180379648eed9b219 P ecededcd854842d4867fda9fa63a4b66 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240506 15:15:31.602675 24157 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:31.598109 24168 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:31.544800 24177 raft_consensus.cc:504] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.550230 24151 raft_consensus.cc:504] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.621141 24172 raft_consensus.cc:3037] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
W20240506 15:15:31.644905 24136 log.cc:924] Time spent T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 1.040s user 0.002s sys 0.004s
I20240506 15:15:31.670257 24051 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.670025 24157 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.671458 23884 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.671664 24192 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:31.675014 24143 leader_election.cc:288] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:31.641111 24166 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:31.586915 24163 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20240506 15:15:31.675668 24036 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.676818 24154 leader_election.cc:288] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:31.681104 24134 log.cc:924] Time spent T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 1.091s user 0.006s sys 0.013s
W20240506 15:15:31.682528 24194 log.cc:924] Time spent T e81c09e0400045b180379648eed9b219 P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20240506 15:15:31.684630 24122 leader_election.cc:288] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.638767 24087 raft_consensus.cc:729] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:31.697809 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
W20240506 15:15:31.698956 24171 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20240506 15:15:31.702917 24157 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.714747 20877 raft_consensus.cc:1261] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240506 15:15:31.824705 20877 pending_rounds.cc:76] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 4
I20240506 15:15:31.825095 20877 pending_rounds.cc:97] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240506 15:15:31.825428 20877 raft_consensus.cc:2866] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:31.651924 24196 raft_consensus.cc:2726] T 815d66b0c2c24db2aa336b62f17837a1 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240506 15:15:31.719054 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 83636.9us
I20240506 15:15:31.723943 20875 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:31.725114 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 105050us
I20240506 15:15:31.688175 24173 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:31.863324 20875 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.664517 23857 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.872166 21026 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240506 15:15:31.872959 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.891922 20924 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:31.894604 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:31.897632 21690 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:31.895190 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.905231 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.917160 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:31.917737 21032 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:31.693039 24168 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.893090 24201 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:31.892063 24157 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:31.921228 20996 leader_election.cc:398] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:31.897528 23662 raft_consensus.cc:482] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:31.922124 24174 raft_consensus.cc:482] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.922441 24151 leader_election.cc:288] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.924103 23857 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.723073 24192 raft_consensus.cc:504] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.924455 24200 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240506 15:15:31.939121 24174 raft_consensus.cc:504] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.925433 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:31.940546 23765 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.941429 24174 leader_election.cc:288] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.896545 24016 raft_consensus.cc:482] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.926945 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.933095 20996 leader_election.cc:401] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240506 15:15:31.934234 20950 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:31.942997 24168 leader_election.cc:288] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.937170 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7c34467d5a4c759c46b85eccf0d47d" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:31.938143 24177 leader_election.cc:288] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
W20240506 15:15:31.926440 24029 consensus_meta.cc:287] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e: Time spent flushing consensus metadata: real 0.872s user 0.010s sys 0.000s
I20240506 15:15:31.944897 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.943744 20953 raft_consensus.cc:2454] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 9.
I20240506 15:15:31.953262 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.891513 24173 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.956691 24173 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.915855 24087 consensus_queue.cc:260] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.957780 24087 raft_consensus.cc:3037] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:31.956279 24029 raft_consensus.cc:504] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.960667 24007 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:31.960877 24029 leader_election.cc:288] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:31.899005 24171 raft_consensus.cc:482] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.963338 24171 raft_consensus.cc:504] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.964434 24051 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:31.964880 24051 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.964792 23884 consensus_queue.cc:1035] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240506 15:15:31.965698 24171 leader_election.cc:288] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:31.970389 21690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:31.899269 24122 raft_consensus.cc:2726] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 15:15:31.966984 24007 raft_consensus.cc:504] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
W20240506 15:15:31.976389 24161 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20240506 15:15:31.902645 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:31.980190 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:31.981799 21030 raft_consensus.cc:2379] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 11: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:31.984787 24007 leader_election.cc:288] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.984944 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:31.984788 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:31.988718 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.989897 21026 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240506 15:15:31.989929 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0693482e35a64dfc90d3944c134718da" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:31.993233 20883 raft_consensus.cc:2454] T 0693482e35a64dfc90d3944c134718da P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:31.993453 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcd32ac1e314101bae9c062a76798df" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:31.994098 20882 raft_consensus.cc:3037] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:31.934541 24154 raft_consensus.cc:482] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:31.996666 24154 raft_consensus.cc:504] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:31.998730 24154 leader_election.cc:288] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:31.933820 24175 raft_consensus.cc:482] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:31.999538 24175 raft_consensus.cc:504] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.001751 24175 leader_election.cc:288] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.002889 24192 leader_election.cc:288] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:31.997150 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:32.004163 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:31.949761 23662 raft_consensus.cc:504] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.004698 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.004916 21026 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:31.965371 24051 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:32.006647 20882 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:32.010226 24153 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.010934 24153 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:31.944161 24201 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.012758 24204 raft_consensus.cc:482] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.013545 23765 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.013387 24204 raft_consensus.cc:504] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.014055 23765 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.014688 24201 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.015918 23765 leader_election.cc:288] T ab1516eeb0ab46f4b471c209c399130f P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.016638 21023 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 11.
I20240506 15:15:32.012962 24086 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.017300 24086 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:32.017846 24152 consensus_meta.cc:287] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.924s user 0.010s sys 0.000s
I20240506 15:15:32.018661 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.016192 24204 leader_election.cc:288] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.016335 24007 raft_consensus.cc:482] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.019730 20880 raft_consensus.cc:1261] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Refusing update from remote peer 7c8800c6dcaf4b6e8dd58c4149f73c4e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240506 15:15:32.028069 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:32.020888 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.023165 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75831e0251ec4bec9c8f3ca6c4a047f0" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.029444 20952 raft_consensus.cc:1226] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 6. Current term is 7. Ops: []
I20240506 15:15:32.018640 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.024863 20949 raft_consensus.cc:1226] T 81525cec16e140db99cd0ef9f2717778 P ecededcd854842d4867fda9fa63a4b66 [term 6 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:32.028748 21027 raft_consensus.cc:3032] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:32.022059 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.030591 20875 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:15:32.030700 21027 raft_consensus.cc:729] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:32.030187 20953 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:32.028380 20880 pending_rounds.cc:76] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Aborting all ops after (but not including) 5
I20240506 15:15:31.979629 21032 raft_consensus.cc:2454] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:32.031893 20880 pending_rounds.cc:97] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240506 15:15:32.032223 20880 raft_consensus.cc:2866] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240506 15:15:32.035282 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.036840 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.019373 21686 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 11.
I20240506 15:15:32.031159 20951 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 7.
I20240506 15:15:32.039652 20949 raft_consensus.cc:1261] T 7e7c34467d5a4c759c46b85eccf0d47d P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:32.041280 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfbe39391094dce8178f175dfb05165" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:32.044368 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ce6e0da3e465f875171ac339f3b50" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.045593 20996 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:32.045562 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10845b6e0cc149f5a5e40605a2ea7e49" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.038198 24210 raft_consensus.cc:482] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.047799 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.043876 21027 consensus_queue.cc:260] T 208b7cbf36514c419696de9c14606c16 P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.048187 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05b4289a334b34a5f571e608bc9191" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.049057 20949 raft_consensus.cc:2379] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 6: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:32.052557 24093 raft_consensus.cc:3037] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:32.052871 20949 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240506 15:15:32.045516 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.059195 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.058184 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f4d07ea30646d0805b63bb641358be" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.049340 24189 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:32.044095 24007 raft_consensus.cc:504] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.062745 24189 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.064276 24007 leader_election.cc:288] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.064824 24189 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.063197 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81702cae5854497d947832321f954762" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:31.949553 24157 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.068542 24157 leader_election.cc:288] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.066162 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.069697 20949 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:32.072129 24153 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.072963 20879 raft_consensus.cc:1261] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240506 15:15:32.072827 24205 consensus_queue.cc:1046] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20240506 15:15:32.074517 21690 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:32.074533 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:32.076534 20953 raft_consensus.cc:3037] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:32.067979 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.082759 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.084620 20922 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:32.084831 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.085052 21030 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P 7610624a16b44896b6aacea1928ab20e [term 8 LEADER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:32.085175 21032 raft_consensus.cc:1261] T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240506 15:15:32.085918 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.086622 20875 raft_consensus.cc:2454] T 74536b2ff8754c03a925c7b726433266 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 10.
I20240506 15:15:32.087553 21654 leader_election.cc:302] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.088935 21688 raft_consensus.cc:2379] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 6: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:32.086009 20998 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:31.981083 24016 raft_consensus.cc:504] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.096055 21026 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:32.038748 24163 raft_consensus.cc:2726] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545) with higher term. Message: Invalid argument: T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7610624a16b44896b6aacea1928ab20e for earlier term 8. Current term is 9.
I20240506 15:15:32.106788 20874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.050376 23662 leader_election.cc:288] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:32.110641 20998 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.111068 20846 leader_election.cc:302] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.105978 21688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.114760 21688 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:32.112228 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.116689 21026 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 11.
I20240506 15:15:32.117727 21653 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters:
I20240506 15:15:32.094144 24157 raft_consensus.cc:2726] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.122318 20998 leader_election.cc:302] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.100107 24196 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 15:15:32.108139 23993 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240506 15:15:32.126137 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.133419 20882 raft_consensus.cc:2379] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 7: Already voted for candidate 00fe71c1675441379f5f2c7490f9e63a in this term.
I20240506 15:15:32.125576 20953 raft_consensus.cc:2454] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 7.
I20240506 15:15:32.128886 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.130177 20878 raft_consensus.cc:2396] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240506 15:15:32.134958 20875 raft_consensus.cc:2454] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 9.
I20240506 15:15:32.135383 20846 leader_election.cc:302] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.135457 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.134044 20845 leader_election.cc:302] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters: ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.099777 24200 consensus_queue.cc:1046] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20240506 15:15:32.048121 24210 raft_consensus.cc:504] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.108626 24030 consensus_queue.cc:1035] T 0693482e35a64dfc90d3944c134718da P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240506 15:15:32.156155 20846 leader_election.cc:302] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.196023 20953 raft_consensus.cc:2454] T 91da71d5ebf84ae7844608806c0d9a94 P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 8.
I20240506 15:15:32.120185 24161 consensus_queue.cc:1035] T 7e7c34467d5a4c759c46b85eccf0d47d P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
W20240506 15:15:32.211382 20846 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 55362.6us
I20240506 15:15:32.120457 24143 raft_consensus.cc:2726] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240506 15:15:32.119623 24171 raft_consensus.cc:3037] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:32.216361 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
W20240506 15:15:32.215611 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 73602us
I20240506 15:15:32.131757 20924 leader_election.cc:302] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:32.218216 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 86975.3us
I20240506 15:15:32.216383 20875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.218881 20924 leader_election.cc:302] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.124135 23662 raft_consensus.cc:2726] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240506 15:15:32.218561 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 84627.8us
I20240506 15:15:32.155918 24120 raft_consensus.cc:2726] T a9f4d07ea30646d0805b63bb641358be P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:32.221980 20924 leader_election.cc:302] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.130745 24029 raft_consensus.cc:2726] T d85358bc4d3f42d897b4c2868c83c8ec P 7610624a16b44896b6aacea1928ab20e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 15:15:32.159868 24199 raft_consensus.cc:482] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 15:15:32.171928 24203 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240506 15:15:32.222582 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.129545 24036 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240506 15:15:32.139257 23993 raft_consensus.cc:482] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:32.228996 20924 leader_election.cc:302] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.124552 24093 raft_consensus.cc:482] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:32.252053 20996 leader_election.cc:302] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.211009 24086 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.234025 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:32.217056 24173 raft_consensus.cc:2726] T 81702cae5854497d947832321f954762 P 00fe71c1675441379f5f2c7490f9e63a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 15:15:32.217090 24180 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.011s sys 0.034s
I20240506 15:15:32.235826 20998 leader_election.cc:302] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.236238 24197 raft_consensus.cc:2781] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:32.223711 24152 raft_consensus.cc:2781] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:32.239992 20845 leader_election.cc:302] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.147634 24118 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.085s
I20240506 15:15:32.280390 24051 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.224462 24007 raft_consensus.cc:2781] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader election won for term 9
I20240506 15:15:32.281750 24007 raft_consensus.cc:686] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [term 9 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:32.282934 20957 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 11.
I20240506 15:15:32.282670 24007 consensus_queue.cc:237] T 0fcd32ac1e314101bae9c062a76798df P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.264305 23884 raft_consensus.cc:2781] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:32.283756 23884 raft_consensus.cc:482] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:32.284132 23884 raft_consensus.cc:3037] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
W20240506 15:15:32.284876 20845 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 45014.7us
I20240506 15:15:32.285650 20845 leader_election.cc:302] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.286943 24161 raft_consensus.cc:2726] T 10845b6e0cc149f5a5e40605a2ea7e49 P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 15:15:32.287074 20845 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:32.287597 24161 raft_consensus.cc:2726] T dd8ce6e0da3e465f875171ac339f3b50 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.288221 24161 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.263463 24093 raft_consensus.cc:504] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.258822 24192 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20240506 15:15:32.290524 24093 leader_election.cc:288] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.291044 20958 raft_consensus.cc:1226] T e7268b79c9094189823e8d245b8c579a P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240506 15:15:32.259507 24016 leader_election.cc:288] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.253931 20846 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:32.272318 24180 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.791s user 0.011s sys 0.035s
I20240506 15:15:32.294749 24173 raft_consensus.cc:2781] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader election won for term 7
W20240506 15:15:32.295260 24183 log.cc:924] Time spent T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.819s user 0.004s sys 0.011s
I20240506 15:15:32.295343 24173 raft_consensus.cc:686] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:32.240922 24199 raft_consensus.cc:504] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.296542 24173 consensus_queue.cc:237] T 208b7cbf36514c419696de9c14606c16 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.297894 24199 leader_election.cc:288] T b96eaaaebde34390bee0dafdfb156080 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.261811 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.233929 24122 raft_consensus.cc:482] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.269575 24147 raft_consensus.cc:2726] T f207fe580fd74c83b4540a552300d5b8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.233893 24168 consensus_queue.cc:1046] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240506 15:15:32.272038 20874 raft_consensus.cc:2454] T f85eeb0daa004acc898342a81ba6fd8d P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 8.
I20240506 15:15:32.237628 24036 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 15:15:32.226714 24169 consensus_queue.cc:1035] T b51ffbf114654032ba2038e989eec2bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20240506 15:15:32.314405 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:32.308633 20845 leader_election.cc:302] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:32.252140 23993 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:32.318054 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91da71d5ebf84ae7844608806c0d9a94" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.320716 21656 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:32.320725 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf3a7304753470787006e0ba7cf8852" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.317526 20878 raft_consensus.cc:1261] T 286345f896bc43ec975183896aee147c P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Refusing update from remote peer 7610624a16b44896b6aacea1928ab20e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240506 15:15:32.317727 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae00aa7ed8440db954bef879396844e" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:32.314414 21026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.325354 21030 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 10. Current term is 11. Ops: []
I20240506 15:15:32.272053 24197 raft_consensus.cc:482] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.329989 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.333250 24199 raft_consensus.cc:2781] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:32.336885 21032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.328183 24177 raft_consensus.cc:2781] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader election won for term 6
I20240506 15:15:32.341398 24168 raft_consensus.cc:3032] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:32.282701 24051 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Requested vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.355288 24199 raft_consensus.cc:482] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.356659 24199 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:32.358209 24177 raft_consensus.cc:686] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [term 6 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:32.365460 20950 raft_consensus.cc:3032] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Stepping down as leader of term 5
I20240506 15:15:32.366144 20950 raft_consensus.cc:729] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:32.312222 24036 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Advancing to term 12
I20240506 15:15:32.310387 24152 raft_consensus.cc:686] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [term 9 LEADER]: Becoming Leader. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:32.352339 21689 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 11.
I20240506 15:15:32.367195 20950 consensus_queue.cc:260] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.385023 20950 raft_consensus.cc:3037] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:32.387493 20922 leader_election.cc:302] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.389600 20845 leader_election.cc:302] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
W20240506 15:15:32.378701 24203 log.cc:924] Time spent T f5a9099e10624a769e45882e9b864930 P 00fe71c1675441379f5f2c7490f9e63a: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240506 15:15:32.355676 24197 raft_consensus.cc:3037] T 74536b2ff8754c03a925c7b726433266 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Advancing to term 10
I20240506 15:15:32.359454 24177 consensus_queue.cc:237] T edfbe39391094dce8178f175dfb05165 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.366086 24093 raft_consensus.cc:482] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:32.365630 24168 raft_consensus.cc:729] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:32.310681 24122 raft_consensus.cc:504] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.324123 24166 consensus_queue.cc:1035] T 801b36bb14184241921c1b97377314fb P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20240506 15:15:32.406535 21656 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:32.392470 24204 raft_consensus.cc:2781] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 11 FOLLOWER]: Leader election won for term 11
W20240506 15:15:32.422590 20921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.131:42913, user_credentials={real_user=slave}} blocked reactor thread for 188109us
I20240506 15:15:32.421469 20950 raft_consensus.cc:2454] T edfbe39391094dce8178f175dfb05165 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 6.
I20240506 15:15:32.429718 20921 leader_election.cc:302] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:32.432969 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
W20240506 15:15:32.435009 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 174667us
I20240506 15:15:32.410167 24189 raft_consensus.cc:2781] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:32.410338 24118 raft_consensus.cc:2726] T faf3a7304753470787006e0ba7cf8852 P 00fe71c1675441379f5f2c7490f9e63a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240506 15:15:32.453100 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 217312us
I20240506 15:15:32.455446 20996 leader_election.cc:302] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66; no voters:
I20240506 15:15:32.460604 20881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 2011) took 1302 ms. Trace:
I20240506 15:15:32.461107 20881 rpcz_store.cc:276] 0506 15:15:31.149674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:31.149880 (+ 206us) service_pool.cc:225] Handling call
0506 15:15:31.151053 (+ 1173us) raft_consensus.cc:1457] Updating replica for 1 ops
0506 15:15:31.353688 (+202635us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 15:15:31.353721 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0506 15:15:31.370197 (+ 16476us) log.cc:842] Serialized 26 byte log entry
0506 15:15:31.565356 (+195159us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b1800455980. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f6314266661 00007f63142733e1 00007f630b3a6eb5 00007f63141e0939 00007f630b3c442b 00007f630b3c282f 00007f630b3c4992 00007f630df704eb 00007f630df75f93 00007f630df76341 00007f630e2c381f
0506 15:15:31.777174 (+211818us) raft_consensus.cc:1631] Marking committed up to 5
0506 15:15:31.778698 (+ 1524us) log.cc:842] Serialized 14 byte log entry
0506 15:15:31.892772 (+114074us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:31.892834 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 15:15:32.431831 (+538997us) raft_consensus.cc:1671] finished
0506 15:15:32.451314 (+ 19483us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 15:15:32.451606 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":343503,"prepare.queue_time_us":1143,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":315330,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":13343,"spinlock_wait_cycles":10483456,"thread_start_us":211561,"threads_started":1,"wal-append.queue_time_us":659493}
I20240506 15:15:32.388911 24199 raft_consensus.cc:504] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.448987 24196 raft_consensus.cc:3037] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 5 FOLLOWER]: Advancing to term 6
I20240506 15:15:32.444190 24210 leader_election.cc:288] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:32.449286 24204 raft_consensus.cc:686] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [term 11 LEADER]: Becoming Leader. State: Replica: ecededcd854842d4867fda9fa63a4b66, State: Running, Role: LEADER
I20240506 15:15:32.413142 24093 raft_consensus.cc:504] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.486254 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe2d6a721f4eaeb4b3f4b5f71e1a91" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.412984 24120 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20240506 15:15:32.468012 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e60758583b247c2946eb30ea2681286" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:32.467911 24147 raft_consensus.cc:2781] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 15:15:32.501518 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.498512 20955 raft_consensus.cc:2379] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00fe71c1675441379f5f2c7490f9e63a in current term 7: Already voted for candidate ecededcd854842d4867fda9fa63a4b66 in this term.
I20240506 15:15:32.493933 24192 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.504989 24192 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.506916 24192 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545)
I20240506 15:15:32.546846 20998 leader_election.cc:302] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.461356 24152 consensus_queue.cc:237] T 481a6073b18543d58315ee2fe4522232 P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.500222 24147 raft_consensus.cc:482] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.475518 24168 consensus_queue.cc:260] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.559514 24168 raft_consensus.cc:3037] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:32.560144 24199 leader_election.cc:288] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.524441 24200 raft_consensus.cc:2726] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.521095 24166 raft_consensus.cc:482] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:32.582659 20846 leader_election.cc:302] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e
I20240506 15:15:32.583200 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cea534ce0498f89b8551ba0b7e5d8" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.593242 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:32.597519 24151 raft_consensus.cc:482] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.598048 24151 raft_consensus.cc:504] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.601714 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d9cec20d4d219e474797b7115172" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.603808 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e3786df1a41928afd8b70061d732a" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.602450 20954 raft_consensus.cc:2454] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 10.
I20240506 15:15:32.577234 24147 raft_consensus.cc:3037] T 91da71d5ebf84ae7844608806c0d9a94 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Advancing to term 9
I20240506 15:15:32.610024 24151 leader_election.cc:288] T 809ee3c2d683400da1f7a753fea4a4a3 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.605755 24087 raft_consensus.cc:2726] T 7e60758583b247c2946eb30ea2681286 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.612426 24204 consensus_queue.cc:237] T 5b648042628a4aec948d269392eb5cf5 P ecededcd854842d4867fda9fa63a4b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.614727 23993 raft_consensus.cc:504] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.617949 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.618736 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.618685 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96eaaaebde34390bee0dafdfb156080" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.622236 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66"
I20240506 15:15:32.580781 24171 raft_consensus.cc:2726] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:32.622970 20952 raft_consensus.cc:3037] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Advancing to term 8
I20240506 15:15:32.624783 24087 raft_consensus.cc:482] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:32.625036 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.626390 23993 leader_election.cc:288] T 5b648042628a4aec948d269392eb5cf5 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 12 election: Requested vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:32.626611 24153 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20240506 15:15:32.580011 24016 raft_consensus.cc:482] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.652676 24016 raft_consensus.cc:504] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.655145 24016 leader_election.cc:288] T 2ca3241be4714e97a24f66b7c1e9c4d8 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.658640 20846 leader_election.cc:302] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.661124 24172 raft_consensus.cc:3037] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Advancing to term 7
I20240506 15:15:32.662912 24086 consensus_queue.cc:1046] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240506 15:15:32.663089 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.664191 24086 raft_consensus.cc:3032] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Stepping down as leader of term 6
I20240506 15:15:32.664695 24086 raft_consensus.cc:729] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7c8800c6dcaf4b6e8dd58c4149f73c4e, State: Running, Role: LEADER
W20240506 15:15:32.665585 20924 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.67.129:44145 timed out after 1.979s (SENT) [suppressed 11 similar messages]
I20240506 15:15:32.671959 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.628325 20877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50468 (request call id 2017) took 1018 ms. Trace:
I20240506 15:15:32.629103 24093 leader_election.cc:288] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.577677 24186 raft_consensus.cc:2726] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240506 15:15:32.646971 21031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:32.627043 24219 log.cc:924] Time spent T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20240506 15:15:32.627955 24122 leader_election.cc:288] T 1a88ec9e65b24b64be49384639f4f9f9 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.668519 24173 raft_consensus.cc:482] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.671855 24087 raft_consensus.cc:504] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.668062 24151 raft_consensus.cc:482] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:32.584817 24001 consensus_queue.cc:1035] T 286345f896bc43ec975183896aee147c P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.066s
I20240506 15:15:32.678803 20877 rpcz_store.cc:276] 0506 15:15:31.609550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:31.609766 (+ 216us) service_pool.cc:225] Handling call
0506 15:15:31.610207 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0506 15:15:31.825985 (+215778us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 15:15:32.628027 (+802042us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54001473d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f01a1 00007f630e2eebee 00007f6313231089 00007f630c913694 00007f630c913622 00007f630c913551 00007f630c9134cc 00007f630c912782 00007f630c716bcc 00007f630c716006
0506 15:15:32.628233 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239150976}
I20240506 15:15:32.684468 23766 raft_consensus.cc:482] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 00fe71c1675441379f5f2c7490f9e63a)
I20240506 15:15:32.681756 20922 leader_election.cc:302] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:32.696019 24094 raft_consensus.cc:3032] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Stepping down as leader of term 10
I20240506 15:15:32.696225 24159 raft_consensus.cc:482] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.698473 21031 raft_consensus.cc:1226] T dae00aa7ed8440db954bef879396844e P 7610624a16b44896b6aacea1928ab20e [term 11 FOLLOWER]: Rejecting Update request from peer 00fe71c1675441379f5f2c7490f9e63a for earlier term 10. Current term is 11. Ops: []
W20240506 15:15:32.757555 20922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.132:34421, user_credentials={real_user=slave}} blocked reactor thread for 75874.7us
W20240506 15:15:32.759696 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 353256us
W20240506 15:15:32.759799 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 248384us
I20240506 15:15:32.593946 24166 raft_consensus.cc:504] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.776907 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6abe388354e4b7e8f0656dda82078bc" candidate_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:32.632048 23884 raft_consensus.cc:504] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.780512 20881 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 10.
I20240506 15:15:32.785250 20879 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:32.772135 23766 raft_consensus.cc:504] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.768734 24151 raft_consensus.cc:504] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.788991 23766 leader_election.cc:288] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:32.774739 24161 consensus_queue.cc:1046] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20240506 15:15:32.790621 20954 raft_consensus.cc:1226] T bc05b4289a334b34a5f571e608bc9191 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Rejecting Update request from peer 7c8800c6dcaf4b6e8dd58c4149f73c4e for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:32.774888 24159 raft_consensus.cc:504] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.768647 24094 raft_consensus.cc:729] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 00fe71c1675441379f5f2c7490f9e63a, State: Running, Role: LEADER
I20240506 15:15:32.793155 24159 leader_election.cc:288] T 9b0fa8a9db084a51a74caa1382efc1c5 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.793292 24094 consensus_queue.cc:260] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.794539 24094 raft_consensus.cc:3037] T dae00aa7ed8440db954bef879396844e P 00fe71c1675441379f5f2c7490f9e63a [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:32.798144 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a9099e10624a769e45882e9b864930" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.798550 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74536b2ff8754c03a925c7b726433266" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.800841 21024 raft_consensus.cc:2379] T 74536b2ff8754c03a925c7b726433266 P 7610624a16b44896b6aacea1928ab20e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 10: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:32.773651 24122 raft_consensus.cc:2781] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 15:15:32.804714 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85eeb0daa004acc898342a81ba6fd8d" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.805187 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208b7cbf36514c419696de9c14606c16" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.811880 20956 raft_consensus.cc:1226] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 9. Current term is 10. Ops: []
I20240506 15:15:32.665699 24086 consensus_queue.cc:260] T 93fe2d6a721f4eaeb4b3f4b5f71e1a91 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.836282 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1516eeb0ab46f4b471c209c399130f" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.841022 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.856824 21030 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecededcd854842d4867fda9fa63a4b66 in term 7.
I20240506 15:15:32.837879 21656 leader_election.cc:302] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7c8800c6dcaf4b6e8dd58c4149f73c4e; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:32.839913 20952 raft_consensus.cc:2454] T f6abe388354e4b7e8f0656dda82078bc P ecededcd854842d4867fda9fa63a4b66 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 8.
I20240506 15:15:32.841979 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:32.858610 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 199250us
I20240506 15:15:32.858779 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
W20240506 15:15:32.842725 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 44474.8us
I20240506 15:15:32.768875 24173 raft_consensus.cc:504] T 35b78f712a194a51abc96c8500a9a611 P 00fe71c1675441379f5f2c7490f9e63a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:32.800292 24231 raft_consensus.cc:482] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.807639 24122 raft_consensus.cc:482] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7c8800c6dcaf4b6e8dd58c4149f73c4e)
I20240506 15:15:32.812705 21024 raft_consensus.cc:2379] T f85eeb0daa004acc898342a81ba6fd8d P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecededcd854842d4867fda9fa63a4b66 in current term 8: Already voted for candidate 7610624a16b44896b6aacea1928ab20e in this term.
I20240506 15:15:32.873337 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:32.873582 21028 raft_consensus.cc:1226] T edfbe39391094dce8178f175dfb05165 P 7610624a16b44896b6aacea1928ab20e [term 6 FOLLOWER]: Rejecting Update request from peer ecededcd854842d4867fda9fa63a4b66 for earlier term 5. Current term is 6. Ops: []
I20240506 15:15:32.850139 24196 raft_consensus.cc:2726] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 15:15:32.876480 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a88ec9e65b24b64be49384639f4f9f9" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.805879 24220 consensus_queue.cc:1035] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20240506 15:15:32.882241 20953 raft_consensus.cc:2454] T b96eaaaebde34390bee0dafdfb156080 P ecededcd854842d4867fda9fa63a4b66 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8800c6dcaf4b6e8dd58c4149f73c4e in term 6.
I20240506 15:15:32.884775 20998 leader_election.cc:302] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
W20240506 15:15:32.952922 20998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 68276.2us
I20240506 15:15:32.954833 20998 leader_election.cc:302] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e; no voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:32.876960 24093 raft_consensus.cc:482] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.923375 20924 leader_election.cc:302] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, ecededcd854842d4867fda9fa63a4b66; no voters:
W20240506 15:15:32.915518 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.130:40545, user_credentials={real_user=slave}} blocked reactor thread for 48441.6us
W20240506 15:15:32.968537 20924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 45254.3us
I20240506 15:15:32.915302 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:32.870867 24232 raft_consensus.cc:2726] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.941051 24171 raft_consensus.cc:482] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:32.873927 24122 raft_consensus.cc:3037] T cc90d9cec20d4d219e474797b7115172 P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:32.953356 24185 raft_consensus.cc:2781] T 7e7c34467d5a4c759c46b85eccf0d47d P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 15:15:32.953866 24077 raft_consensus.cc:2726] T 805cea534ce0498f89b8551ba0b7e5d8 P 7610624a16b44896b6aacea1928ab20e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 15:15:32.880427 24199 consensus_queue.cc:1046] T bc05b4289a334b34a5f571e608bc9191 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20240506 15:15:33.013070 20921 leader_election.cc:302] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:32.981190 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:32.972429 24190 raft_consensus.cc:2781] T f85eeb0daa004acc898342a81ba6fd8d P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 15:15:33.006908 24236 raft_consensus.cc:2726] T e47e3786df1a41928afd8b70061d732a P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 15:15:32.880901 24016 raft_consensus.cc:482] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:33.021358 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ee3c2d683400da1f7a753fea4a4a3" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:33.008708 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:33.021538 20958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54b354e91941efb28b61b312923d41" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.983320 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:33.027871 21024 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 9.
W20240506 15:15:32.985033 24036 consensus_meta.cc:287] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e: Time spent flushing consensus metadata: real 0.596s user 0.006s sys 0.005s
I20240506 15:15:33.030825 20921 leader_election.cc:302] T f5a9099e10624a769e45882e9b864930 P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e
I20240506 15:15:33.034761 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e"
I20240506 15:15:33.035406 24237 raft_consensus.cc:2726] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 15:15:33.031836 20845 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:33.033825 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286345f896bc43ec975183896aee147c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:33.039077 20954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:33.039713 20954 raft_consensus.cc:2454] T fc2acc72bba8443fb8d836e09497cb3c P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fe71c1675441379f5f2c7490f9e63a in term 10.
I20240506 15:15:33.039793 20949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51ffbf114654032ba2038e989eec2bc" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:33.039988 20846 leader_election.cc:302] T 6b54b354e91941efb28b61b312923d41 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a; no voters: 7610624a16b44896b6aacea1928ab20e, ecededcd854842d4867fda9fa63a4b66
I20240506 15:15:33.040872 20958 raft_consensus.cc:1261] T 208b7cbf36514c419696de9c14606c16 P ecededcd854842d4867fda9fa63a4b66 [term 7 FOLLOWER]: Refusing update from remote peer 00fe71c1675441379f5f2c7490f9e63a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240506 15:15:33.043776 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2acc72bba8443fb8d836e09497cb3c" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a"
I20240506 15:15:32.981602 24087 leader_election.cc:288] T fc2acc72bba8443fb8d836e09497cb3c P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:32.980341 24151 leader_election.cc:288] T 5f7a6eab190f4de2bbd9e6b2ff60cdc1 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
I20240506 15:15:33.045616 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0fa8a9db084a51a74caa1382efc1c5" candidate_uuid: "ecededcd854842d4867fda9fa63a4b66" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "00fe71c1675441379f5f2c7490f9e63a" is_pre_election: true
I20240506 15:15:33.008574 24166 leader_election.cc:288] T b51ffbf114654032ba2038e989eec2bc P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:33.053020 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3241be4714e97a24f66b7c1e9c4d8" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ecededcd854842d4867fda9fa63a4b66" is_pre_election: true
I20240506 15:15:32.969156 24093 raft_consensus.cc:504] T c89eb38afe3b43719809589e0aeb7eef P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:33.011633 23884 leader_election.cc:288] T 286345f896bc43ec975183896aee147c P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 10 election: Requested vote from peers 00fe71c1675441379f5f2c7490f9e63a (127.19.67.129:44145), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:33.057106 21025 raft_consensus.cc:1261] T 0fcd32ac1e314101bae9c062a76798df P 7610624a16b44896b6aacea1928ab20e [term 9 FOLLOWER]: Refusing update from remote peer ecededcd854842d4867fda9fa63a4b66: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240506 15:15:33.057230 20921 leader_election.cc:302] T 6d096daea6fa4799a608705b3dc732db P ecededcd854842d4867fda9fa63a4b66 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7610624a16b44896b6aacea1928ab20e, 7c8800c6dcaf4b6e8dd58c4149f73c4e, ecededcd854842d4867fda9fa63a4b66; no voters: 00fe71c1675441379f5f2c7490f9e63a
I20240506 15:15:33.064356 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b648042628a4aec948d269392eb5cf5" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e"
I20240506 15:15:33.191134 21686 raft_consensus.cc:3037] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 11 FOLLOWER]: Advancing to term 12
W20240506 15:15:33.204563 20950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.000s sys 0.012s
W20240506 15:15:33.205040 20950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s user 0.000s sys 0.012s
I20240506 15:15:33.207652 21683 raft_consensus.cc:1226] T f6abe388354e4b7e8f0656dda82078bc P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 8 FOLLOWER]: Rejecting Update request from peer 7610624a16b44896b6aacea1928ab20e for earlier term 7. Current term is 8. Ops: []
W20240506 15:15:33.210258 24244 log.cc:924] Time spent T 801b36bb14184241921c1b97377314fb P 7610624a16b44896b6aacea1928ab20e: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20240506 15:15:33.212498 24236 consensus_queue.cc:1046] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240506 15:15:33.213645 24236 raft_consensus.cc:3032] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Stepping down as leader of term 7
I20240506 15:15:33.214047 24236 raft_consensus.cc:729] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7610624a16b44896b6aacea1928ab20e, State: Running, Role: LEADER
I20240506 15:15:33.214190 24166 raft_consensus.cc:482] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:33.214809 24166 raft_consensus.cc:504] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:33.215579 24236 consensus_queue.cc:260] T f6abe388354e4b7e8f0656dda82078bc P 7610624a16b44896b6aacea1928ab20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } attrs { replace: true } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:33.216929 24166 leader_election.cc:288] T 8884787c4c4b474685c7e903e8789e90 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913)
I20240506 15:15:33.217726 21686 raft_consensus.cc:2454] T 5b648042628a4aec948d269392eb5cf5 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 12.
I20240506 15:15:33.214262 21023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49620 (request call id 514) took 1347 ms. Trace:
I20240506 15:15:33.220988 21023 rpcz_store.cc:276] 0506 15:15:31.866928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 15:15:31.867110 (+ 182us) service_pool.cc:225] Handling call
0506 15:15:33.214083 (+1346973us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540015f4d4. stack: 00007f630b392ba3 00007f630b392aa0 00007f630acf7584 00007f6314a0d9e4 00007f6314a0d979 00007f630e2746b1 00007f630e2f34fe 00007f63132319c7 00007f630c915c14 00007f630c915ba2 00007f630c915ad1 00007f630c915a4c 00007f630c914d02 00007f630c716bcc 00007f630c716006 00007f630c7159af
0506 15:15:33.214187 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271372032}
I20240506 15:15:33.223485 24189 raft_consensus.cc:482] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7610624a16b44896b6aacea1928ab20e)
I20240506 15:15:33.223996 24189 raft_consensus.cc:3037] T 0693482e35a64dfc90d3944c134718da P ecededcd854842d4867fda9fa63a4b66 [term 10 FOLLOWER]: Advancing to term 11
I20240506 15:15:33.227187 24255 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:33.227816 24255 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
W20240506 15:15:33.228555 21656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.67.129:44145, user_credentials={real_user=slave}} blocked reactor thread for 355831us
I20240506 15:15:33.230718 24255 leader_election.cc:288] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 00fe71c1675441379f5f2c7490f9e63a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ecededcd854842d4867fda9fa63a4b66 (127.19.67.130:40545), 7610624a16b44896b6aacea1928ab20e (127.19.67.131:42913), 7c8800c6dcaf4b6e8dd58c4149f73c4e (127.19.67.132:34421)
W20240506 15:15:33.233036 24168 consensus_meta.cc:287] T f207fe580fd74c83b4540a552300d5b8 P 00fe71c1675441379f5f2c7490f9e63a: Time spent flushing consensus metadata: real 0.673s user 0.008s sys 0.003s
I20240506 15:15:33.195050 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7a6eab190f4de2bbd9e6b2ff60cdc1" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:33.199910 24143 raft_consensus.cc:482] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ecededcd854842d4867fda9fa63a4b66)
I20240506 15:15:33.235006 24143 raft_consensus.cc:504] T 75831e0251ec4bec9c8f3ca6c4a047f0 P 7610624a16b44896b6aacea1928ab20e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } attrs { replace: true } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:33.241230 20998 leader_election.cc:302] T fc2acc72bba8443fb8d836e09497cb3c P 7610624a16b44896b6aacea1928ab20e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fe71c1675441379f5f2c7490f9e63a, 7610624a16b44896b6aacea1928ab20e; no voters:
I20240506 15:15:33.245635 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b78f712a194a51abc96c8500a9a611" candidate_uuid: "00fe71c1675441379f5f2c7490f9e63a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "7610624a16b44896b6aacea1928ab20e" is_pre_election: true
I20240506 15:15:33.008847 24171 raft_consensus.cc:504] T e81c09e0400045b180379648eed9b219 P 00fe71c1675441379f5f2c7490f9e63a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:33.034440 24016 raft_consensus.cc:504] T ab1516eeb0ab46f4b471c209c399130f P 00fe71c1675441379f5f2c7490f9e63a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } }
I20240506 15:15:33.121879 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d096daea6fa4799a608705b3dc732db" candidate_uuid: "7610624a16b44896b6aacea1928ab20e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" is_pre_election: true
I20240506 15:15:33.250357 21689 raft_consensus.cc:2454] T 6d096daea6fa4799a608705b3dc732db P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7610624a16b44896b6aacea1928ab20e in term 7.
I20240506 15:15:33.042582 24174 raft_consensus.cc:482] T e7268b79c9094189823e8d245b8c579a P 00fe71c1675441379f5f2c7490f9e63a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 15:15:33.050338 24036 raft_consensus.cc:504] T cc90d9cec20d4d219e474797b7115172 P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "7c8800c6dcaf4b6e8dd58c4149f73c4e" member_type: VOTER last_known_addr { host: "127.19.67.132" port: 34421 } attrs { promote: false } }
I20240506 15:15:32.944149 24235 consensus_queue.cc:1035] T 6b54b354e91941efb28b61b312923d41 P 7610624a16b44896b6aacea1928ab20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
I20240506 15:15:32.880667 24231 raft_consensus.cc:504] T a9f4d07ea30646d0805b63bb641358be P 7c8800c6dcaf4b6e8dd58c4149f73c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00fe71c1675441379f5f2c7490f9e63a" member_type: VOTER last_known_addr { host: "127.19.67.129" port: 44145 } attrs { replace: true } } peers { permanent_uuid: "ecededcd854842d4867fda9fa63a4b66" member_type: VOTER last_known_addr { host: "127.19.67.130" port: 40545 } } peers { permanent_uuid: "7610624a16b44896b6aacea1928ab20e" member_type: VOTER last_known_addr { host: "127.19.67.131" port: 42913 } } peers { permanent_uuid: "